the effect of efqm self-assessments on action …...as the purpose of the efqm model is to help...

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014

The effect of EFQM self-assessments on action research performance: Lessons on how to improve the efficiency and effectiveness of research on New Public Management Petter Øgland

Department of Informatics, University of Oslo, Norway

Abstract. New Public Management (NPM) is based on the idea that public sector

organisations can be made more effective by adopting management perspectives and methods

more similar to those used in the private sector, but the economic and managerial rationality

of NPM often conflicts with practitioner rationality, causing challenges in the implementation

of NPM. Action research could be relevant for studying NPM implementation by searching

for solutions that are seen as satisfying from both for the managerial and the practitioner

perspective, but action research is a demanding type of research with high risk of failure. On

the other hand, if one believes that NPM will improve organisational performance, then

perhaps it will also improve action research performance. The aim of this study is to

investigate whether the efficiency and effectiveness of action research can be improved by

using the NPM logic of the EFQM self-assessment method. The question is theoretically

investigated through the perspective of double-loop learning and empirically tested for eleven

months in the context of using canonical action research (CAR) for implementing NPM in a

Norwegian public sector organisation. Despite challenges in establishing and sustaining the

CAR process, the EFQM approach proves helpful. Contributions to theory and practice is

summarised in how the effect of EFQM self-assessments contributes in improving single-loop

learning, double-loop learning and deutero-learning in action research.

Keywords: New public management, EFQM self-assessments, total quality management,

double-loop learning, canonical action research.

1. Introduction

The aim of new public management (NPM) is to improve public sector cost-efficiency by

changing how public sector organisations operate (Hood, 1991). Implementing NPM is not

easy. Professionals working within organisations where NPM is being implemented

sometimes complain that the NPM view on improvement brings along a type of economic and

managerial logic that is conflicting with practitioner logic. Perspectives like NPM tend to

introduce the need for procedures, documentation and measurements in a manner that stifles

the creativity, flexibility and general attitudes that are necessary for making the professional

perform at an optimum level. In support of the practitioner view, Fukuyama (2013) argues

that NPM-like programmes have focused too much on measuring results and too little on

measuring competence and autonomy. Similar views have also been argued by organisational

and management scholars earlier, e.g. Mintzberg (1996).

One way of developing knowledge about how to implement NPM in a manner that is not only

acceptable by the practitioner community but also supports the logic of the practitioners and

helps develop competence and autonomy is the method of action research. Writing

specifically about the issue of using action research for cultivating local knowledge

(practitioner knowledge) in the context of the economic and managerial knowledge developed

through the use of means like business process reengineering (BPR) and total quality

management (TQM), McNiff (2000) makes account of action research success in the context

of organisational learning. She also explains some of the challenges of dealing with power


and control when the researcher becomes part of organisational politics. As pointed out by

Simonsen (2009), action research is a demanding type of research with high risk of failure.

However, as the purpose of introducing NPM was to make the organisation more effective

and efficient from an economic and managerial perspective, what would be the effect of

applying a NPM-like logic for designing and managing the action research project? While

many action researchers would be reluctant to try such an effort as the practitioner logic of the

action researcher crashes in similar way with the economic and managerial logic of NPM-like

efforts like TQM, it would nevertheless be interesting to study the effects of TQM on action

research in a more controllable environment provided by doing self-assessments.

Action research in the form articulated by Kurt Lewin in the 1940s was a type of engineering

science used for researching strategies for social change (Gold, 1999, p. 295). Today there

are many forms of action research. Canonical action research (CAR) is one of several action

research variants that have been developed within the information systems (IS) community

for the purpose of researching organisational change in the context of IS (Davison et al,

2004). As IS plays an important role in implementing NPM infrastructure, CAR could be an

interesting type of action research to study. A further interesting aspect of CAR is that it is

argued to be robust through a set of five principles that have been developed as a consequence

of analysing numerous cases of IS action research failure. The principles CAR are related to

(1) researcher-client agreement, (2) cyclical process model, (3) theory, (4) change through

action, and (5) learning through reflection.

Even though the principles may provide a sound way of making sure that the action research

produces reliable and valid results, it may not always be an easy task to make sure that the

criteria associated with the CAR principles are being met. According to Clark (1972), action

research involves interplay between problem owners, practitioners (action researchers) and

research audiences. When Øgland (2014) looks at the problem of developing CAR

researcher-client agreements in politically challenging environments, he makes use of game

models describing the relationship between clients (problem owners), researchers

(practitioners) and the scholarly community (research audiences). Although understanding

the researcher-client agreement game is important for succeeding with CAR, the researcher-

client agreement is only one of the five principles. Getting a good researcher-client

agreement without aligning with the other principles is not expected to result in sustainable

and effective CAR.

Studying the effect of TQM on action research in real-life practice requires having an action

research process to investigate. Within the director-general’s IT staff at the Norwegian Tax

Administration (NTAX) there has been a challenge on how to improve the practice of IT

Governance since a reorganisation in 2008. It has been recognised both from within NTAX

and by external consultants hired by NTAX (DNV, 2005; Gartner, 2010) that there is a vital

need for improving ITG, but at the same time it has also been recognized both from within

and without that the NTAX culture may be the organisations greatest enemy in the sense that

it prevents conventional step by step methods of implementing ITG. In November 2013 there

were initial discussions about starting an action research process for looking into the situation.

As part of the feasibility study, the EFQM assessment model (Oakland, 1999) was used for

diagnosing the organisational readiness for trying to improve ITG through the use of CAR.

The paper is structured into six sections. After having motivated the research and presented

the main hypothesis in this introductory section, the next section is a select review of


literature on action research and the EFQM assessment model seen from the perspective of

double loop learning. This is followed by a methodology section where it is explained how

the theoretical models are to be empirically investigated through the use of a real-world

experiment. Analysis of the outcome of the experiment is presented in the fourth section.

This is followed by a fifth section that discusses the analysis of results in the context of

related work. The final section summarises the study by identifying contributions to theory

and practice plus directions for further research.

2. Literature review

The purpose of the literature is to develop an action research strategy where the EFQM self-

assessment model has been integrated with the action research methodology. The action

research strategy will be explained within the context of what Argyris and Schön (1978) refer

to as double-loop learning.

2.1 How to implement action research as double-loop learning

There are studies on using action research to improve organisational EFQM scores (e.g.

Prybutok & Ramasesh, 2005). There are also action research studies where the EFQM model

has been integrated in the research design. For example, Kontostavlaki et al (2010) refer to

experience in using EFQM as a foundation for action research in the context of assessing and

evaluating environmental programmes at school with the aim of increasing the environmental

awareness and sensitivity among the students. Nevertheless, no literature has been found on

how to use the EFQM model for improving action research.

This does not mean that the action research community sees no need for quality improvement.

Boog et al (2008) have written specifically about the need for quality improvement of action

research, but their focus has been on the ethics and moral standards that are used as governing

variables in action research rather than the standards, methods and models of TQM. On the

other hand, according to Lilford et al (2003) there is little difference between action research

for organisational change and TQM. Both action research and TQM are cyclical activities

involving examination of existing processes, change, monitoring the apparent effects of the

change and further change. Both emphasise active participation of stakeholders. The

examples used to illustrate action research would serve equally well as examples of TQM and

vice versa, they claim.

When considering the structure and nature of the EFQM model, the argument made by

Lilford et al can be made more explicit. Action research is often structured as a cyclic

process. The so-called RADAR logic of EFQM is a four-step cyclical ordering of activities

involving specifying results (R), planning and developing approaches (A), deploying

approaches (D), assessing and refining (A&R). In action research there is examination of

existing processes and monitoring of effects on change after having designed interventions

according to some theory of change. Examination of existing processes in the EFQM context

is done by comparing the organisation with the five enabler criteria of the EFQM model. The

effects of change are monitored through the lens of the four results criteria of the model. The

theory of change is embedded in the model by how the results criteria and enabler criteria are

linked at a sub-criterion level represented by a 32 by 32 logical matrix. If one compares this

structure with a structured version of action research, like canonical action research (CAR),

there is a strong match between action research and TQM, just as Lilford et al suggest.

As the purpose of the EFQM model is to help organisations reflect both on what they are

doing and how well they are doing it, double-loop learning (Argyris and Schön, 1978) can be


a useful perspective when considering how to integrate TQM and action research in the

context of organisational change. The theory of double-loop learning was developed from the

perspective that people and organisations often have a tendency for changing without

questioning whether they are focusing on the right issues to change. Although learning and

improvement without questioning governing variables may result in increased efficiency, the

increased efficiency may be related to things that do not matter or matter in a negative way.

As illustrated in figure 1, change in efficiency by designing and implementing action

strategies should be guided by considerations on effectiveness in terms of reflecting on the

governing variables.

Figure 1. Model of how learning happens in practice (Argyris et al, 1985, p. 84)

For the purpose of researching double-loop learning, Argyris et al (1985) have suggested a

particular type of action research they refer to as “action science”. The reason for doing so is

because they felt that too much action research lacked theoretical focus. Systems theory,

control theory, decision theory and the critical theory of the Frankfurt school exemplify

theories of action (ibid, chapter 1). When applying this kind of theory, there are two types of

empirical hypothesis to consider, they say, the first type is of the form “Agent a has

disposition d” and the second type has the form “Action (or pattern of actions) a will lead to

(be causally responsible for) consequences c” (ibid, p. 55). They also articulate the latter

formula as “in situation s, to achieve a consequence c, do action a” (ibid, p. 81).

As the motivational problem in this study has to do with the implementation of IS strategies

for ensuring the development of NPM, canonical action research (CAR) is a more useful form

of action research in the context of the IS community. Nevertheless, CAR and action science

have much in common. For example, CAR hypotheses are expected to be formulated along

the lines of “in situation S with salient features F, G and H, the outcomes X, Y, Z are

expected from actions A, B and C” (Davison et al, 2004, p. 74).

What this means in the context of the model in figure 1 is that the action research hypothesis

is a causal statement dealing with governing variables, action strategy and consequences.

Single-loop learning deals with the relationship between actions A, B and C and the

consequences X, Y and Z when salient features F, G and H of the situation S (governing

variables) are being left unquestioned. Double-loop learning deals with reflection about the

situation S with salient features F, G and H in relation to what has been observed. In other

words, single-loop learning is learning about the efficiency of the action strategy. If the

action strategy was a strategy for developing NPM, single-loop learning would be concerned

with issues like the time and cost for implementing NPM using this strategy. Double-loop

learning, on the other hand, is concerned with issues like why one would be interested in

NPM, whether the action strategy is based on a proper understanding of the situation S and

whether the salient features F, G and H are the most useful features for describing S.


Another important issue in double-loop learning theory is the issue of how to implement

double-loop and single-loop learning. The motivational position taken by Argyris and Schön

(1978) is that organisations and individuals are generally good at single-loop learning as this

usually means adapting to the norms of the organisation while they are generally bad at

double-loop learning as this means questioning norms and culture. Other scholars, for

example Lyytinen and Robey (1999), are more negative in terms of pointing out how cultural

norms and politics may often hinder even single-loop learning. In the context of

implementing information systems (IS), not only do organisations fail to learn when

introducing IS, the IS development process becomes an acceptable means for explaining

failure. Organisations learn to fail.

Brunsson and Jacobsson (2000) have a similar view on organisational learning in the context

of TQM. As they see it, problems like implementing double-loop learning cannot be

understood in the context of looking at the organisation in isolation. Each organisation is

tangled up in a political network of suppliers, customers, management consultants, quality

standards and competitors. Often it is easier to explain why organisations are not learning

through the use of TQM by considering who the important actors are. The outcome can be

predicted by looking at what these actors have to gain or loose by complying with TQM

standards as compared to pretending to comply. Brunsson and Jacobson believe that the

market economy results in fake TQM in the sense that the most important actors in the TQM

game (organisations, management consultants, TQM auditors, makers of TQM models,

methods and standards) benefit from organisations being seen to comply with TQM even

when they do not necessarily do so.

The model in figure 1 is useful for conceptualising action research about NPM

implementation as an action strategy. As mentioned above, it is quite likely that the

consequences of the action strategy will be that nothing happens. According to the logic of

the diagram, this may be cause for reflection about the governing variables that have been

used for designing the action research. McKay and Marshall (2001) have argued the need for

action research to be designed in such a double-loop manner to make sure that not only does

the research result in learning about the client problem but also about effective research

designs. Looking at the same problem within the context of software process improvement,

Øgland (2007) argues that quality standards used for process improvement also define the

nature of the kind of scholarly knowledge that needs to be developed. Integrating the EFQM

assessment method with action research is an extension of this idea.

Hypothesis 1. The EFQM self-assessment method is useful for learning about how to

implement action research as double-loop learning.

.

The hypothesis above was motivated by the ideas that action research can be made more

effective when viewed from the perspective of the action (process of organisational change)

and research (producing new and relevant knowledge) being seen through a lens of double-

loop learning when both processes are being linked through the use of a quality standard like

EFQM. The next two sections of the literature review will focus more specifically on what it

means to integrate EFQM and action research through the double-loop learning model in the

context of making action research effective and efficient.


2.2 How to make action research effective

An example of an ineffective NPM action research would be a type of NPM implementation

study that focuses on issues that are irrelevant for the purpose of implementing NPM. For

example, if the action researcher questions the NPM ideology and focuses his research on

explaining resistance to change based on the oppressive nature of NPM, he may produce an

interesting case study, but it will be ineffective as action research. It is ineffective action

research because the governing variables in the action research strategy are not aligned with

what the client is trying to achieve in terms of implementing NPM. On the other hand, if the

researcher is working for a client who wants to increase workplace democracy and reduce

social injustice, a critical perspective on how the organisation is implementing NPM might be

instrumental for making the action research effective.

As TQM assessment models like the EFQM model are used for identifying the aims of an

organisation, how it is performing with reference to such aims, and what kind of methods it is

using to achieve the aims, assessing the action research against such TQM standards should in

principle be useful for developing double-loop learning and increased effectiveness.

An overview of the EFQM model is presented in figure 2. The model consists of nine criteria

that are further divided into a set of 32 sub-criteria. As indicated by the diagram, the first five

criteria on the left are described as enables as they are used for evaluating issues like

leadership, people management, policy and strategy, resource management and process

management. The final four criteria on the right are termed results criteria and are used for

evaluating organisational performance through a “balanced scorecard” of people results,

customer results, impact on society, and key performance business results.

Figure 2. EFQM model with weights and relationships between criteria

When measuring an organisation as a result of doing an EFQM assessment, the organisation

is given a score between 0 and 1000 points. The percentages associated with each of the

criteria in the diagram above are used as weights. According to EFQM theory it is assumed

that different criteria have different levels of impact on overall excellence. The diagram

presents the weights as they were defined in 1999.

Although the EFQM model has been developed for assessing all kinds of organisations,

universities and industrial research and development (R&D) departments sometimes argue

that the research process is so fundamentally different from the manufacturing process that


they question the validity of the TQM approach (Endres, 1997, p. 6). However, after

struggling with how to interpret the language of the TQM assessment models in the context of

R&D, Endres (1997, pp. 125-137) summarise industrial experience as positive. The TQM

models help the R&D departments assess their current level of performance and structure

long-term strategies for improvement. The EFQM model has also been extensively used

among universities and institutions of higher education (Hides et al, 2004).

The idea of using the EFQM model for improving the effectiveness of action research can be

seen as an extension of how TQM assessment models have been used in industrial R&D units,

assuming that action research and engineering research are sufficiently similar to allow this

extension to work. With reference to how Argyris et al (1985) saw systems theory, control

theory, decision theory and critical theory as special cases of action theory, action research

and engineering research could be seen as tightly related.

Hypothesis 2. Applying EFQM self-assessments on the action research process will increase

the effectiveness of action research.

2.3 How to make action research efficient

One way of characterising efficient research is to focus on features like low cycle times from

initiation to publication of research, low cost in conducting research, getting good peer

feedback in review and having the work widely cited after publication. While some would

argue that ethics and moral awareness are important quality characteristics of action research

(e.g. Boog et al, 2008), in the context of CAR it is more natural to see such aspects as part of

the effectiveness of the approach rather than the efficiency. However, if the action research is

carried out in a politically challenging environment, the duration of how long the action

researcher is able to survive without selling out or getting fired could be used as an efficiency

measurement.

From a TQM perspective, increase in efficiency is achieved through the means of process

improvement. Figure 3 gives a process perspective of CAR. The CAR process is cyclic and

is made up of a sequence of five sub-processes for diagnosis, action planning, intervention,

evaluation and reflection. The CAR process starts by entering the point of diagnosis, then

cycles through the five steps as many times as needed, and exits after then final round of

reflection has been completed.

Figure 3. CAR process model (Davison et al, 2004)

When considering how to integrate the EFQM model with the CAR process, there are several

possibilities. One approach could be to apply the EFQM model at a meta-level by black-

boxing the CAR process and using the RADAR logic of the EFQM model for investigating


results, approaches, deployment, assessments and refinements in the way the CAR process is

being implemented. Another approach would be to integrate the EFQM and CAR models at

the same level by using the EFQM assessment method as the key tool at the evaluation stage

of the CAR process.

The approach used in this study consists of updating and using the results from EFQM

assessments while going through the steps of the CAR process. What this means is that CAR

and EFQM assessments are carried out in parallel through a sequence of five steps. The first

step is the CAR diagnosis of NPM situation together with an EFQM assessment of the CAR

situation. The second step is action planning for treating the NPM situation together with

action planning for improving the CAR situation. The third step is to carry out the CAR

intervention while recording the quality of the intervention through the use of the EFQM

enabler criteria. The fourth step is to interpret the outcome of the CAR intervention while

also assessing the EFQM results criteria. The fifth step is to consider the interpretation of

CAR outcomes in terms of what to do next while also looking at the overall revised EFQM

assessment score by comparing it with the reference EFQM score from the diagnosis stage.

The chosen method of integrating the EFQM model with the CAR process has relevance for

effectiveness in the sense that areas for improvement are “automatically” selected each time

the CAR process goes through the action planning stage. The improvement process is driven

by the EFQM results criteria and how the results criteria are weighted. This means that

changes in focus are adjusted in a manner to increase efficiency in relation to the governing

variables that are already in existence. In other words, the approach is expected to have an

impact on effectiveness but the main drive is a concern for improving efficiency.

In terms of defining governing variables for action research, much has been written about

how to improve research efficiency without mentioning TQM. When Endres (1997)

discusses the use of TQM for improving research and development (R&D) from a general

perspective, he recommends the reduction of cycle time as goal for research units starting on

the TQM journey. When translating this into the context of academic action research, cycle

time could refer to the time between the initiation and publication of research.

Another relevant goal could be to reduce the costs (man-hours) for writing and publishing

scientific papers while making sure they get published in journals and conference proceedings

of high merit. Some universities have classification systems where academic outlets are

graded on scales like high level (2 points), normal level (1 point), and low level (0 points).

Issues like citations, h-indexes and rankings of scholars are also relevant. The Google

Scholar system is particularly useful because it automatically places results in a global

context.

Oakland (2003) gives a general introduction on how business measurements and processes in

general can be analysed and improved through a TQM perspective. Whether the business

process consists of selling sausages or publishing results from scientific studies does not

matter in the context of TQM.

In addition to methods for evaluating research, there has to be methods for enabling research.

There is extensive literature on how to do action research and how to write and publish

academic papers. Huff (1999) writes about scholarly writing for publication. Novak and

Gowin (1984) write about research designs and formats of scholarly papers. Fisher and Ury

(1981) have written about negotiations in a context that could be relevant for establishing and


maintaining the research-client agreement in CAR. As the EFQM method of improving

action research efficiency puts focus on a few parameters at a time, literature reviews have to

be designed for figuring out how to improve the parameter in question.

Hypothesis 3. Applying EFQM self-assessments on the action research process is an optimal

strategy for improving action research efficiency.

3. Research methodology

When trying to understand the impact of the EFQM assessment model on canonical action

research (CAR) through real-world experiments, the mode of research is that of a CAR

practitioner studying how to improve his own practice or what Donald Schön (1983) refers to

as reflective practice. To make use of reflective practice as a research design, the approach

taken in this study is the self-improvement action research methodology described by McNiff

and Whitehead (2006).

3.1 Population and sampling procedure

The target population in the study is the action research (AR) community in general and the

canonical action research (CAR) community in particular. There are no assumptions about

geography, age, gender and culture when defining the target population, but it is assumed that

the action research and organisational change is carried out in a politically challenging

organisation such as a public sector bureaucracy. The sampling procedure consists of the

researcher using himself as a single-unit sample from the AR/CAR practitioner population.

3.2 Research instruments and measures

Due to the nature of reflective practice research, the researcher himself is the main research

instrument. It is through the researcher that data is being collected, analysed and interpreted.

However, parts of the data will be collected through the use of the EFQM assessment tool.

This will be done in two different ways. At the diagnosis stage at the beginning of the CAR

cycle, existing CAR practice will be evaluated by assessing each of the 32 sub-criteria of the

EFQM model on a scale from 0% (unable) to 100% (excellent). The scores reflect subjective

interpretations with the average score of 50% is used to represent “I don’t know”. From the

stage of action taking and onwards, the RADAR diagrams in the tables 1 and 2 below will be

used for reassessing selected sub-criteria that were found particularly relevant during the

diagnosis and action planning stages.

Relevance &

usability

Guidance Unable

to

demonst

rate

Limited

ability

to

demonst

rate

Able to

demonst

rate

Fully

able to

demonst

rate

Recogni

sed as

global

role

model

Scope &

relevance

A coherent set of results, including key

results, are identified that demonstrate

the performance of the organisation in

terms of its strategy, objectives and the

needs and expectations of the relevant

stakeholders.

X

Integrity Results are timely, reliable and

accurate.

X

Segmentation Results are properly segmented to

provide meaningful insight.

X

Performance

Trends Positive trends or sustained good X


performance over at least 3 years.

Targets Relevant targets are set and consistently

achieved for the key results, in line with

the strategic goals.

X

Comparisons Relevant external comparisons are

made and are favourable for key results,

in line with the strategic goals.

X

Confidence There is confidence that the

performance levels will be sustained

into the future, based on established

cause & effect relationships.

X

Scale 0% 25% 50% 75% 100%

Overall score 43%

Table 1. Example of RADAR evaluation of a results sub-criterion

The use of the RADAR method for evaluating a results sub-criterion is illustrated in table 1

by setting an X in each of the seven parameters to be evaluated and calculating the overall

score as the average of the percentage value given in the scale column. In case the subjective

assessments are supported by objective data some of the seven parameters may be evaluated

on a continuous scale. The RADAR evaluation of enabler criteria is exemplified in table 2.

Approach Guidance Unable

to

demonst

rate

Limited

ability

to

demonst

rate

Able to

demonst

rate

Fully

able to

demonst

rate

Recogni

sed as

global

role

model

Sound The approaches have a clear rationale,

based on the relevant stakeholder needs,

and are process based.

X

Integrated The approaches support strategy and are

linked to other approaches as

appropriate.

X

Deployment

Implemented The approaches are implemented in

relevant areas, in a timely manner.

X

Structured The execution is structured and enables

flexibility and organisational agility.

X

Assessment

& refinement

Measurement The effectiveness and efficiency of the

approaches and their deployment are

appropriately measured.

X

Learning &

creativity

Learning and creativity is used to

generate opportunities for improvement

or innovation.

X

Improvement

& innovation

Outputs from measurement, learning

and creativity are used to evaluate,

prioritise and implement improvements

and innovations.

X

Scale 0% 25% 50% 75% 100%

Overall score 7%

Table 2. Example of RADAR evaluation of an enabler sub-criterion

The reliability and validity of the EFQM assessments are not expected to be high as part of

the challenge of the research is to figure out how to use the EFQM model in the context of

CAR.


3.3 Data collection

Data will be collected while traversing one single cycle of CAR. This means that data will be

collected by describing the process of diagnosis, action planning, taking action, evaluation

and reflection while trying to aid the development of NPM in a public sector organisation. In

addition to the qualitative and quantitative description of process and process outcome at the

various stages of the cycle, numerical data will be produced through EFQM assessments.

Some of these assessments will be based on objective measurements, but to a large extent

they are dependent on subjective interpretations.

3.4 Data analysis

The causal relationship common to all the three hypotheses is represented in figure 4 by a

causal arrow in the upper part of the diagram and moderator arrows in the lower part. The

causal arrow suggests that higher levels of maturity of the CAR strategy measured by the

EFQM enabler criteria result in higher levels of CAR success measured by the EFQM results

criteria. The moderator arrows suggest that that the environmental counterstrategies moderate

the impact in the sense that a higher degree of EFQM enabler maturity is necessary to achieve

a comparable level of EFQM results in environments that are more politically complex.

Figure 4. Model of causal relationships in the self-improvement process

As the EFQM data are expected to have low reliability and validity, the mode of analysis will

be that of observing the outcome of assessing the EFQM enablers and results before and after

the intervention and then try to explain the meaning of the numbers rather than assuming that

they can provide any conclusive evidence by themselves.

4. Analysis of results

The analysis of results is structured by following the five steps of the action research process

model and reflecting on the impact of the EFQM assessment model for each step.

4.1 Diagnosis

In 2008 the Norwegian Directorate of Taxes (Skattedirektoratet, SKD) decided to reorganise

the IT function by redesigning the traditional IT department and other service functions as a

single unit (Skatteetatens IT- og servicepartner, SITS) that was still owned by SKD but was

conceptually externalised and managed as though it were an external unit. The model in

figure 5 shows how service level agreements (SLA) between SKD and SITS are used for

daily supply-chain management while IT staff monitors and interacts with SITS from an

information technology governance (ITG) perspective.

The redesign at the Norwegian tax administration (NTAX) was based on strategic reasoning

and not a response to IT failures. In fact, the use of IT at NTAX had been an ongoing success

story and occasionally described as at role model for other Norwegian public sector

organisations. However, there were also challenges along different dimensions, including the

maintaining of a balance between user involvement from the professional bureaucracy and

allowing the users to get “optimal solutions” without thinking about costs and complexity for

Environmental

counterstrategies

CAR outcome assessed by

EFQM results criteria

CAR strategy assessed by

EFQM enabler criteria


the organisation as a whole (Wroldsen, 2008). Viewed from the perspective of the

professional bureaucracy, or rather the machine bureaucracy of functionaries interacting

between the professional bureaucracy and the IT department, there was discontent in how

they had become totally dependent on decisions made by the IT department and thus had

difficulty in seeing themselves as the real owners of the IT solutions. This was pointed out in

an external study (Statskonsult, 2002). In the study it was further argued, based on the theory

of five organisational structures (Mintzberg, 1983), that ITG run by a machine bureaucracy in

the hands of the adhocracy of technological development may not be a good solution in the

long run.

Figure 5. Organisational ITG strategy at the Norwegian tax administration

From the viewpoint of new public management (NPM), the model in figure 5 can be seen to

incorporate all of the three fundamental NPM ideas of competition, contracts and control.

Having the IT department and the other service departments conceptually externalised could

be the first step towards making systematic use of the market. The SLA illustrates the use of

contractual management, and by having SITS develop an ISO 9001-based quality

management system to be audited by SKD (e.g. IT staff) illustrates the NPM idea of control.

However, even though SITS has been trying to develop a quality management system based

on ISO 9000 standards, and there are explicit agreements between SKD and SITS on how

quality audits should be performed, there has been no progress in the development of an ISO

9001 audit process.

Furthermore, the reason why there has been no progress is not because SKD has been

perfectly happy with SITS performance. The latter part of a study on total quality

management (TQM) implementation at NTAX dealing with the post-2008 period produced

findings in support of the idea that quality management in certain processes, such as the

COBOL software quality, was not only becoming more difficult to monitor from the SKD

perspective but that the compliance with NTAX standards and procedures had gotten worse

after the 2008 reorganisation (Øgland, 2013).

The EFQM diagnosis in the table below represents the researcher’s subjective understanding

of the situation above from the viewpoint of wanting to assist in the process of developing an

ISO 9001 quality audit system to be run from IT staff as support in monitoring the SITS

quality management system. This viewpoint means that EFQM result criteria like “customer

results” refers to the relationship between the researcher and the client (head of IT staff),

“people results” refers to how the research team evaluate their own performance and

satisfaction, “society results” refers to remaining parts of NTAX, and “key performance

results” refers to published research related to the development of an audit system. All the

five enabler criteria refer to the management of the action research process from the research

perspective. The table is sorted by having the criteria in greatest need of improvement come

out on top.

CEO

IT staff

SKD SITS SLA

Audit


EFQM Criterion Reverse weight (RW) Score% Priority = RW * Score%

6. Customer results 5,0 25 1,25

4. Partnership and resources 11,1 25 2,78

9. Key performance results 6,7 50 3,35

5. Processes 7,1 50 3,55

7. People results 11,1 38 4,22

1. Leadership 10,0 50 5,00

2. Strategy and planning 12,5 44 5,50

3. People 11,1 50 5,55

8. Society results 16,7 50 8,35

Total 42

Table 3. Total EFQM evaluation November 2013

The benefit of using the EFQM model in this manner is that it makes the action researcher a

part of the system being diagnosed. Although the verbal diagnosis surrounding the diagram

in figure 5 should be as objective as possible, the EFQM self-assessment stresses that the

diagnosis is a model viewed from a certain perspective for the specific purpose of making

interventions. The NTAX diagnosis can be thought of as trying to disclose a political game

that prevents the organisation from designing and developing an ISO 9000 audit programme,

but the nature of the CAR diagnosis is of a different kind.

When using the NPM logic of the EFQM model in figure 2 to diagnose the CAR situation, the

focus becomes managerial and economic in the sense of looking at issues like leadership and

business results. Unlike the NTAX diagnosis, which might be viewed from a game

theoretical perspective as it is concerned with the strategic interplay between several decision

makers, when the action researcher makes use of EFQM self-assessments he is doing this for

the purpose of controlling and improving his own process. Game theory may be one of

several perspectives relevant in such a context, as there are customers and other stakeholders

to deal with, but the overall problem is a control problem. By using the EFQM model to

understand the situation, the result is having the problem framed as an engineering problem.

The CAR process becomes something that can be studied from an industrial engineering

perspective (e.g. Krick, 1962).

As the purpose of the CAR diagnosis is to design control and improvement strategies, it is

more important that the diagnosis captures the characteristics of the situation in a useful

manner than being perfectly accurate. Although the data in the table above are based upon

evaluating all the 32 sub-criteria of the EFQM model, the lack of practical organisational

insights means that the diagnosis should be seen as an initial guess.

4.2 Action planning

The purpose of action planning is to search alternative treatments for the diagnosis, compare

such treatments and decide which one to implement. In the context of wanting to design a

quality management audit system for IT staff to monitor SITS (figure 5), the first question

becomes how to establish a researcher-client agreement (RCA) for getting started with the

practical work and deciding how the practical work should be done.

As was mentioned when commenting on how the EFQM model was used as a diagnosis tool,

the table could be sorted in a manner that indicated the most important criteria to improve.

According to the 80/20 rule (Koch, 1998), it should be sufficient to focus on the two weakest

criteria (~20%) in the table as the majority of problems are often caused by a minority of

causes. The summary diagnostics from table 3 consequently suggest that action planning


should focus on customer results and partnerships/resources. In other words, the initial

usefulness of the EFQM model for connecting the action research stages of diagnosis and

action planning is that the focus of planning follows more or less automatically from the

outcome of the diagnosis.

When using the two highest ranking criteria in the EFQM diagnosis from table 3 as a basis for

planning action in the context of redesigning enablers and getting better results, the enabler

criterion dealing with partnerships can be investigated further by looking at the scores at the

sub-criterion level, but in the case of the result criterion of customer results it is necessary to

consult the EFQM theory explaining how the enabler and results criteria are causally

connected. This illustrates the further usefulness of EFQM for controlling action research.

According to the British Quality Foundation (2000), customer results are caused by

phenomena addressed by ten sub-criteria distributed among all the five enabler criteria of the

EFQM model. In table 4 the initial score for these sub-criteria have been listed and sorted

according to scores and reverse weights in order to make it easier to start the improvement

process by focusing on critical factors with poor performance.

CAF/EFQM Criterion Reverse weight (RW) Score% Priority = RW * Score%

4a Good supplier/partner relationships to satisfy

customer

11,1 25 2,78

5b Improving processes to satisfy customers 7,1 50 3,55

5c Product and service development 7,1 50 3,55

5d Product and service delivery 7,1 50 3,55

5e Customer relationship management 7,1 50 3,55

1c Leaders’ involvement with customers 10,0 50 5,00

2a Establishing customers’ needs and

expectations

12,5 44 5,50

2c Balancing customers’ needs and

expectations

12,5 44 5,50

3b People have the skills and competence to

deal with customers

11,1 50 5,55

3c People’s involvement with customers 11,1 50 5,55

Total 46

Table 4. Ranking of EFQM enablers that should be addressed for improving customer results

Following the 80/20 principles outlined above, the first step of planning is to focus on

developing good supplier/partner relationships (4a) and improving processes to satisfy

customers (5b).

In the context of a research-client agreement, the client (head of IT staff at NTAX) is the

customer and academia is the supplier. Developing good supplier/partner relationships means

to make sure that the client feels comfortable with the action research approach. In this case

the head of IT staff at NTAX told the researcher (member of IT staff) that no research would

be allowed unless external funding was provided for. A first step of action planning was

consequently to discuss with academia how this problem could be solved. A more detailed

account of the political challenges in developing a research-client agreement based on this

story of writing research proposals and applying for funding is found in a study dealing with

RCA-like problems from a game theoretical perspective (Øgland, 2014).

A second step in the part of action planning that dealt with the academic side of the RCA was

to identify academic journals and conferences that could be relevant for producing results and

monitoring progress for the action research process. The conferences included in figure 6 are


restricted to Nordic and European conferences to restrict the travelling budget from becoming

an obstacle.

Figure 6. Relevant conferences for presenting research and meeting fellow action researchers

For most of the conferences in this diagram, papers are submitted during spring and the

conferences themselves are held during autumn. The ideal situation would be to work on

seven separate papers designed specifically for each of the seven conferences, submit each

paper for each conference and go to the conferences where the papers have been accepted. As

there may be overlap between conferences during autumn, it may not be possible to

participate in all, so it may be necessary to select those conferences that are seen as most

useful for the research. Table 5 gives a more detailed description of each of the most relevant

conferences.

When it comes to the sub-criterion 5b ranked as the second most important in table 4, the

researcher asked for being allowed to start the development of an ISO 9001-based NTAX

audit management system to be run by IT staff for the purpose of investigating the

performance of the SITS quality management system. The answer, however, was that no

research and development would be allowed until the issue concerning external funding had

been settled. As a consequence of this, the only customer process of relevance is the

communication between the member of IT staff and the head of IT staff concerning progress

in getting external funding.

The third part of action planning related to the RCA is consequently to develop a process of

communicating progress in getting external funding on a regular basis. As time is spent

reading scholarly literature, discussing with representatives of the scholarly community,

writing research proposals, submitting application for research grants and preparing papers for

journals and conferences in case the applications should be accepted, reporting to NTAX on a

monthly basis through the use of emails seems like a reasonable approach.

To summarise the contribution of the EFQM model at the stage of action planning, what the

EFQM helps to clarify is that planning should be done for increasing the score of all the four

EFQM result criteria. In addition to the technical planning of the quality audit system this

includes planning for client satisfaction and academic success. The EFQM model provides

links between enabler criteria and results criteria that gives instant suggestions on where to

focus the planning in order to achieve results in the most critical domains.

SCIS (submit)

UKSS (submit)

ECIS (submit)

NOKOBIT (submit)

QMOD (submit)

SCIS/IRIS (conference)

UKSS (conference)

QMOD (conference)

NOKOBIT (conference)

NEON (conference)

NEON (submit)

IRIS (submit)

ECIS (conference)

Jan

Apr

Jul

Oct

Feb

Mar

May

Jun Aug

Sep

Nov Dec


Conference title Submit Conference Rejection

rate

Comments

European Conference on

Information Systems (ECIS)

December June High Possibility for meeting fellow IS

and ITG researchers.

Scandinavian Conference on

Information Systems Research

(SCIS)

February August High Possibility for meeting fellow

researchers doing IS action

research on organisations in

Scandinavia.

Quality Management and

Organisations Development

Conference (QMOD)

March August Medium Possibility for meeting

international QMOD researchers.

Norsk Konferanse for

Organisasjoners bruk av

informationsteknologi

(NOKOBIT)

July November Medium Possibility for meeting fellow ITG

researchers and IS action

researchers in Norway.

UK System Society International

Conference (UKSS)

June September Low Possibility for meeting fellow

researchers doing action research

with a focus on systems theory.

Information Systems Research in

Scandinavia (IRIS)

April August Low Possibility for meeting fellow

researchers doing IS action

research on organisations in

Scandinavia.

Nettverk for

organisasjonsforskning i Norge

(NEON)

September November Low Possibility for meeting fellow

Norwegian scholars researching

organisational development

through the use of action research.

Table 5. Overview of relevant conferences for presenting information systems action research

4.3 Execution of action strategy

Although the action planning evolved over several months and is still in a process of

evolving, the principle ideas described in the section above were used as a basis for

formulating an action strategy made up of three basic steps.

(1) Apply for research funding,

(2) prepare papers for academic outlets while waiting for response, and

(3) report progress to NTAX.

As most of the planning was done in November 2013, the action strategy has been in constant

use for eleven months. A chronological view of how the execution of the action strategy is

given in table 6.

Member of IT staff Head of IT staff

2013 Nov

28.11. After having successfully defended his PhD

thesis, the researcher reports back to NTAX and he

asks for permission to continue his work by

maintaining contacts with the academic network

while doing practical TQM-related action research

at NTAX.

29.11. Head of IT staff congratulates with

completing the PhD and he agrees to meet

at NTAX on December 5th

.

2013 Dec 5.12. The researcher meets with a group at NTAX

concerned with tax research and the possibilities

for building networks between NTAX and the

scholarly community. As the head of IT staff was

prevented from meeting, the researcher sends a

mail summarising the meeting with the tax

06.12. Head of IT staff confirms the mail

by saying that a post.doc project sounds

interesting.

13.12. In a meeting at NTAX, the head of

IT staff says the research proposal looks


research group and says that they were, among

other things, discussing the relevance of

investigating the quality function at SITS and

possibilities for doing post.doc research.

12.12. The researcher has discussed with various

people at UiO and outlined a post.doc research

proposal as input for discussion at NTAX the next

day.

20.12. The researcher has discussed the situation

with people at UiO. Their response is that a

contractual collaboration between UiO and NTAX

would be nice, but they have no budget at the

moment. However, the matter will be discussed

further and possible solutions will be outlined.

relevant and interesting but also says that it

may be difficult to defend having a

researcher working in his staff unless a part

of the research is externally funded.

2014 Jan 26.01. There have been discussions at UiO. While

a formal collaboration with NTAX is considered

useful, the lack of budgets makes it impossible to

contribute to the funding of the research. It is

assumed that the situation will be better in 2015,

but for 2014 it is recommended to apply for

external funding outside of UiO. UiO will make

sure that the researcher maintains access to UiO

infrastructure. The researcher asks whether it

would be possible for him to get computer access

at NTAX and start doing the action research in an

informal manner while the formalities are

gradually being cleared out.

31.01. Head of IT staff acknowledges the

problem in getting funding from UiO and

suggests either taking contact with other

universities or abandoning the project by

becoming a regular office clerk at NTAX.

2014 Feb 10.02. After taking up the issue with UiO, the

response from the university is that there should be

no need for burning bridges. If the action research

design for developing an audit system could be

defined as a regular NTAX development project

for 2014, it could be turned into an action research

project in 2015 when the financial situation at UiO

is expected to have improved.

No response.

2014 Mar 07.03. The researcher repeats his requests for

getting an office at NTAX with computer access in

order to start the audit system development,

regardless of whether it will end up as a

development project or action research, but also

explains that he is spending time reading and

trying to figure out how to collaborate with various

people at UiO who might become important

collaborators once the contract gets settled.

No response.

2014 Apr 03.04. The researcher informs that UiO has

identified a relevant fund, and he has consequently

spent his time since the 10th

of March writing an

application. A draft version of the application is

appended to the mail. The deadline for submitting

is on the 9th

of April and a recommendation from

NTAX is needed as an appendix to the application.

08.04. The researcher reminds NTAX that the

application is due for tomorrow and that he has not

04.04. The head of IT staff says that there

is too much going on at NTAX at the

moment, but he would like to look through

the application. He says that it should be

sent regardless of him being able to read it

or not.

08.04. The head of IT staff writes he is tied

up in meetings, so the application should

be sent without the recommendation.


received the NTAX recommendation paper yet.

2014 May 07.05. The researcher informs that he is spending

the time writing a paper for the UKSS conference

to be submitted by the end of the month while

waiting for the response from the application.

No response.

2014 Jun 06.06. The researcher informs that the UKSS

conference was cancelled, so the paper was

submitted to a journal instead (IJSS). He is now

writing on a paper for the 3rd

international

innovation in information infrastructure workshop

(IIIOS) to be submitted by the end of the month.

No response.

2014 Jul 04.07. The researcher informs that the two papers

have been submitted. He has also submitted

another paper to the NOKOBIT conference and is

working on a journal paper (EJOLTS). However,

he has also been informed that the application was

rejected. He requests permission to rewrite and

resubmit the application based on the feedback

they got. He also reminds the head of IT staff that

it would be useful if he could get a furnished office

and be allowed to start the audit process when the

internal audit group plan their audit of the quality

function at SITS during the autumn.

No response.

2014 Aug 20.08. The researcher informs about a submitted

paper being accepted for presentation at the IIIOS

conference and that a paper for the NEON

conference is in a final stage of development. He

also informs about discussions with UiO on how to

redesign and submit a new research grant

application.

No response.

2014 Sep 15.09. The researcher informs about the

NOKOBIT paper being rejected, the NEON

abstract being accepted, and the current status in

writing the RFFH application.

22.09. The researcher sends a draft version of the

RFFH project proposal for comments and asks for

a confirmation document from NTAX.

No response.

2014 Oct 01.10. As there is no response from NTAX the

researcher sends a reminder.

08.10. As there is still no response from NTAX the

researcher sends another reminder.

10.10. As UiO starts to worry about written NTAX

support to include in the project proposal, the

researcher phones NTAX and is being informed

that a confirmation document in support of the

project is being written.

13.10. The final version of the grant application is

discussed with UiO and submitted to RFFH.

10.10. The head of IT staff emails a

document that gives formal support of the

project and some input on issues that have

to be looked into to make sure research

interests fits with NTAX interests.

Table 6. Execution of the action strategy


The execution of the action strategy matches with the three steps that make up the action

strategy. The first step of the action strategy was to try to get external funding for the

research. In the table it is seen how this theme is brought up in December 2013 and how it

remains a central issue in all of the monthly status reports. The second step was to prepare for

the actual research by reading, experimenting, discussing and writing papers for academic

outlets. Although time was spent reading, writing and discussing from the beginning of the

project in November 2013, it was only by the end of May 2014 that the papers had matured

sufficiently to be ready to be submitted. The third step of the action strategy was to make

sure that the client (head of IT staff) was getting progress reports. The table shows that

progress reports were written at least once a month. The table also illustrates that it was

difficult arranging face-to-face meetings or getting response in general.

The table below gives an account of the scientific papers that have been worked on as part of

the action research effort. For each paper the title of the paper, the time of initiation, and the

motivation for writing the paper have been recorded. As the process continues, control

statistics in terms of size of paper (number of words), cost of writing (minutes of editing) and

cycle time by counting the weeks from initiation to termination are updated.

Scientific paper Start Motivation Size

(words)

Cost

(minutes)

Time

(weeks)

Status

#1 Can a critical

systems approach

compensate for lack

of management

commitment when

implementing TQM-

based organisational

learning?

03.11.2013 Summary of PhD

thesis using

different data and

analysis methods.

13531 7853 33 28.07.2014

IJSS journal

confirms

having

received the

paper for

review

#2 Improving the

effectiveness of IT

governance in public

sector organisations

by bootstrapping

quality control for

supply-chain

management

30.11.2013 Research

proposal

5945 1580 3 13.12.2014

Conditionally

accepted by

SKD: Partial

funding from

UiO required

#3 The principle of

Researcher-Client

Agreement in

Canonical Action

Research

14.12.2013 Getting the RCA

in order

7967 3161 39 25.08.2014

Not accepted

NOKOBIT

conference

(rejection

rate = 38%)

#4 The effect of EFQM

self-assessments on

action research

performance: New

Public Management

in a public sector

financial organisation

15.12.2013 Monitor the CAR

programme

22109 9811 42 10.11.2014

Accepted for

NEON

conference

(session on

governance,

control, and

learning)

#5 IKT i offentlig

forvaltning og New

Public Management:

Pacman-strategi for

forbedring av kvalitet

og sikkerhet i

19.04.2013 Research

proposal (as part

of grant

application)

2832 2223 9 23.06.2014

Rejected by

RFFH


Skatteetatens IKT-

styring #6 Could constructive

empiricism be more

useful than critical

realism for

conducting action

research on

information

infrastructure

development?

02.05.2014 Getting involved

in philosophical

debates

concerning AR

methodology

2589 2102 16 04.08.2014

Accepted for

IIIOS

conference.

(Rejection

rate = 11%).

#7 Canonical Action

Research and

Information Systems

Development

11.05.2014 Drafting a book

for summarising

the prospective

post.doc research.

28686 2973 28 09.11.2014

In progress

(publisher

undecided)

#8 Learning how to

implement Total

Quality Management

through the use of

Pac-Man video game

simulations


with theory of

organisational

change and the

AR practitioner

community

17947 6275 11 07.10.2014

Preparing for

EJOLTS

journal

#9 Empirical

constructivism makes

action research

similar to design

science research


with action

research vs.

design science

debate.

1295 59 16 01.09.2014

Preparing for

ECIS

conference

#10 Title not yet decided. 17.08.2014 Writing a book to

articulate political

aspects of the

research.

450565 9509 9 13.10.204

Preparing for

publication

(Lulu Press)

#11 Mechanism Design

for Total Quality

Management: Using

the Bootstrap

Algorithm for

Changing the Control

Game

21.08.2014 Secondary PhD

supervisor

recommends

having the thesis

published as a

book.

104977 1275 9 18.10.2014

Published

(Lulu Press)

#12 The principle of

Researcher-Client

Agreement in

Canonical Action

Research

25.08.2014 Getting the RCA

in order

7927 3205 5 27.09.2014

Accepted for

NEON

conference

(open track)

#13 New Public

Management and

ICT in public

administration

Using the bootstrap

algorithm for

cultivating IT

governance

02.09.2014 Research

proposal (as part

of grant

application)

4570 1967 7 13.10.2014

Submitted to

RFFH

Average 51611 3999 17

Table 7. Production processes during period of action research (November 2013 – November 2014)

The development of products in the table above lists three research proposals, seven research

papers and three books. As seen from the status column on the far right, the processes for the

research proposals, one of the books and three of the research papers have been completed.

For the remaining two research papers and two books the final outcome is not yet clear.


The five enabler criteria of the EFQM model can be used for assessing the execution of the

action strategy from a TQM point of view. Table 8 shows the results from evaluating the

leadership process as executed by the action researcher with respect to his one-unit cell as an

individual organisation.

Criterion 1: Leadership Score Comments

1a Leaders develop the mission, vision and values and

are role models of a culture of excellence

43 The mission, vision and values of action

research at NTAX is to test and develop

theories of ISO 9001 auditing as explained in

the research proposal. Overall RFFH

external evaluation of research proposal was

3/7 = 43%.

1b Leaders are personally involved in ensuring that

the organisation’s management system is developed,

implemented and continuously improved

50 This paper illustrates the way the action

research is being developed, implemented

and continuously improved.

1c Leaders are involved with customers, partners and

representatives of society

18 The story in table 6 explains how the

researcher (leader) is involved with

customers (NTAX management), partners

(UiO), and representatives of society

(colleagues at NTAX).

1d Leaders motivate, support and recognise the

organisation’s people

50 Self-motivation.

Average score 40

Table 8. Evaluation of leadership

The evaluation of leadership illustrates the difficulties in subjective evaluations when there

are few objective standards to measure against. The comparatively low score for criterion 1c

is due to difficulties with understanding the needs and expectations of the main customer

(head of IT staff). Leadership is difficult to evaluate and it maintains generally difficult to say

whether the current status is dominantly good (>50%) or bad (<50%).

Table 9 shows the results from trying to evaluate the strategy and planning process for the

action research.

Criterion 2: Strategy and planning Score Comments

2a Policy and strategy are based on the present and

future needs and expectations of shareholders.

25 The diagnosis in section 4.1 illustrates an

attempt to understand the stakeholders.

2b Policy and strategy are based on information from

performance measures, research, learning and

creatively related activities.

25 The EFQM assessment model is the main

tool used for understanding capability.

2c Policy and strategy are developed, reviewed and

updated.

18 The strategy is formulated in section 4.2

dealing with action planning.

2d Policy and strategy are deployed through a

framework of key processes.

50 The deployment of strategy is explained in

section 4.3 dealing with execution of action

strategy.

2e Policy and strategy are communicated and

implemented.

50 The implementation of strategy is also

explained in section 4.3 dealing with

execution of action strategy.

Average score 34

Table 9. Evaluation of strategy and planning

In order to influence customer satisfaction, the action planning suggested that criteria 2a and

2c were of particular importance. The updated evaluation reflects that it is difficult to

understand the stakeholders and it is difficult to know whether the process of development,

review and update of strategy is adequate.


The results in table 10 are self-reflections used to evaluate the human resources processes as

practiced by the action researcher as a member of the IT staff at NTAX.

Criterion 3: People Score Comments

3a People resources are planned, managed and

improved.

46 Self-improvement as an action researcher.

Citation indexes, h-indexes and rankings are

relevant for assessing the situation.

3b People’s knowledge and competencies are

identified, developed and sustained.

8 Self-improvement as an action researcher.

3c People are involved and empowered. 18 Same as 1c “leader’s involvement with

customers”.

3d People and the organisation have a dialogue. 11 Limited contact.

3e People are rewarded, recognised and cared for. 39 Being allowed to do research is an award in

itself.

Average score 24

Table 10. Evaluation of people management

Sub-criteria 3b and 3c were identified as part of the action planning process. As people

management was not identified as critical in the diagnosis, the other sub-criteria have been

assessed as “I don’t know” (50%).

Table 11 shows the evaluation of how partnerships and resources are being managed.

Criterion 4: Partnerships and resources Score Comments

4a External partnerships are managed 29 The partnership with UiO is based on

friendship and trust and works well. For

NTAX the partnership has to be made more

formal, but attempts to apply for research

grants have so far resulted in failure.

4b Finances are managed 25 So far NTAX has paid for research expenses

such as conference travels.

4c Buildings, equipment and materials are managed 64 As long as the researcher works from his

home office or UiO, the infrastructure is

sufficient. At NTAX there is an office

without furniture.

4d Technology is managed 50 Technology is managed when the researcher

works from his home office or UiO. At

NTAX there is an office without phone and

computer.

4e Information and knowledge are managed 32 Access to literature is provided by access to

the UiO facilities. The researcher is

incapable of knowing what is happening at

NTAX.

Average score 40

Table 11. Evaluation of partnerships and resources

According to the action planning process, sub-criterion 4a is of critical importance. Although

the partnership between the researcher and the research community (UiO) has been working

fine, NTAX insists of having it formalised. As failure to get external funding may result in

termination of the action research process, the outcome of the RADAR analysis is only 29%.

The next area of concern is process management. As the member of IT staff has been

working for 42 weeks at the cost of 37.5 hours per week, the costs of implementing the audit

system so far has been 1575 man-hours without gaining any practical results although several

scientific papers have been written. As seen from table 7, about 866 of the 1575 man-hours


(55%) have been used for writing. Table 12 shows results from evaluating the processes of

doing action research.

Criterion 5: Processes Score Comments

5a Processes are systematically designed and

managed.

50 The researcher does not know what he is

expected to do at NTAX.

5b Processes are improved, as needed, using

innovation in order to fully satisfy and generate

increasing value for customers and other stakeholders.

11 Process improvement aimed at improving

customer satisfaction is done ad hoc.

5c Products and services are designed and developed

based on customer needs and expectations.

21 Research grant application and proposal

based were on NTAX needs and

expectations. Rejected. Paper written for

NOKOBIT conference was not accepted.

5d Products and services are produced, delivered and

serviced.

36 Research proposals are written and research

papers are submitted.

5e Customer relationships are managed and enhanced. 18 Monthly status reports to NTAX, but little

feedback and no data.

Average score 27

Table 12. Evaluation of processes

According to the action planning process, the sub-criteria 5b, 5c, 5d and 5e are of critical

importance. As the first process sub-criteria 5a was not identified among the 20% most

important sub-criteria in the action planning process, it has been assessed as “I don’t know”

(50%) in the table above.

To summarise this subsection, the usefulness of applying the EFQM model for evaluating the

action strategy is in the way the enabler criteria of the model makes it possible to think of the

single-person management of the action research process as an organisation conceptually

comparable to any type of organisation. In principle it should be possible to learn about how

to improve action research performance by comparing enablers and results for the action

research unit with enablers and results from similar and different types of organisations.

4.2 Evaluation of action strategy outcomes

The main results from executing the action strategy have been explained as part of the

narration of the execution of the action strategy in the previous sub-section. The results can

be summarised in three bullet points corresponding to the three steps of the action strategy.

A research grant application was written and submitted, but ended up being rejected.

Of the six research papers written and submitted, three papers have been accepted, one

paper has been rejected, and two papers are pending.

The monthly progress reports to NTAX have produced some feedback but not much.

As getting funding for the research was necessary for getting the action research going, and

consequently the aim of the intervention, the failure in getting the application accepted means

that the action aspect of the action research has failed. The research aspect of the action

research, on the other hand, has not been a total failure as 50% of the scientific work

submitted for publication has been accepted while the outcome of the remainder is presently

unclear.

As explained when analysing the diagnosis and the planning for action, the general aim of the

action strategy was to improve client satisfaction (“customer results”, EFQM criterion 6) with


reference to the scores in table 3. Reflecting on the execution of the action strategy, table 13

provides a revised assessment of customer results.

Criterion 6: Customer results Score Factor Final Comments

6.1 Perception measurements 21 0,75 16 Research salary in comparison with

industry averages in the private and

public sector could be seen as an indicator

of customer satisfaction.

6.2 Performance measurements 29 0,25 7 The history recorded in table 6 gives the

impression that the client is not against

the idea of doing action research for

improving organisational performance but

is not enthusiastic about the idea either.

Sum 23

Table 13. Evaluation of customer-oriented results

Although getting an average score of 23 for customer satisfaction is worse than the average

score of 25 at the beginning of research, the adjustment should be seen as a consequence of

instrument calibrations rather then process adjustments.

The business logic behind the people results (EFQM criterion 7) is that satisfied people tend

to be better and more stable performers. Table 14 shows the results of using this criterion as a

means for the action researcher to do a self-evaluation both as a formal member of IT staff

and as an informal member of the UiO scholarly community.

Criterion 7: People results Score Factor Final Comments

7.1 Perception measurements 50 0,75 38 The researcher is satisfied with the

flexibility given when being part of IT

staff but is worried about whether being

allowed to do action research.

7.2 Performance measurements 17 0,25 4 The researcher visited UiO twelve times

since the beginning of the process and

NTAX twice (2/12 = 17%).

Sum 42

Table 14. Evaluation of people results

There are minimal changes in people results when comparing the results presented during

diagnosis with the results after the intervention. It is not an important criterion for the

moment.

The business logic of evaluating society results (EFQM criterion 8) is to make sure that the

organisation does not develop an unfortunate social reputation that could have a negative

influence on business performance.

Criterion 8: Social responsibility results Score Factor Final Comments

8.1 Perception measurements 7 0,75 5 As the researcher spends no time at

NTAX he has no understanding of who

his colleagues are and what they might

think.

8.2 Performance measurements 22 0,25 6 The researcher communicates with the

head of IT staff by email by sending

monthly status reports.

Sum 11

Table 15. Evaluation of society results


When translating this criterion into the context of doing action research at NTAX, social

reputation translates into ability to get along with colleagues at NTAX and UiO. There are

minimal changes in society results when comparing the results presented during diagnosis

with the results after the intervention.

The “key performance results” (EFQM criterion 9) in the case of doing action research on the

development of quality audit systems is to produce and publish high quality research. Table

16 shows the results of using the key performance results criterion for evaluating the action

research process from a scholarly perspective.

Criterion 9: Key performance results Score Factor Final Comments

9.1 Perception measurements 25 0,50 13 The researcher has a vague impression

that UiO expects researchers to publish 2-

4 papers of acceptable quality on an

annual basis, but has not yet been able to

confirm what the expectations are.

9.2 Performance measurements 33 0,50 17 Google Scholar statistics results in a

ranking of 186.

Sum 30

Table 16. Evaluation of key performance results

The changes in key performance results are a result of calibrating the measurement

instruments by introducing a performance measurement method based on the Google Scholar

statistics.

To summarise this subsection, the usefulness of applying the EFQM model for observing the

results of executing the action strategy is in the way the result criteria of the model function as

a balanced scorecard. Using the EFQM model makes the action researcher not only focus on

published research on how to develop a quality audit system but also on issues like the RCA

that are necessary for getting sustainable key performance results of high quality.

4.5 Specification of learning

From the action perspective of the action research, the key learning was that the method used

for developing a research proposal and applying for a grant was unsuccessful and should

consequently be analysed and improved. There are several books and internet resources that

deal with the issue of writing research applications (e.g. Berry, 2010; Aldridge & Derrington,

2012; Sternberg, 2013). How scientists get funded is in itself an interesting and relevant topic

for action research and other types of research (Laudel, 2006). Concerning the research

aspects of the action research learning, Øgland (2014) relates this issue to the researcher-

client agreement (RCA) used in CAR and uses this for discussing games and politics in action

research.

The EFQM model can also be used as part of the reflection process. When looking at the

overview in the table below it can be seen that the main characteristics are that the quality of

partnership with UiO and the satisfaction within the action research team have increased

while all other criteria have gotten worse. However, none of the observed changes are

statistically significant.

What this outcome means in terms of learning is that the EFQM model has been used in a too

informal manner. This does not necessarily mean that the study was poorly designed. It

could also mean that this kind of action research designs take time to mature. Exactly how


long it takes before the EFQM measurement instruments become sufficiently tested and

calibrated to be useful for practical research could be a research question in itself.

EFQM 2013 EFQM 2014

Criterion Weight % Points Criterion Weight % Points

1. Leadership 100 50 50,0 1. Leadership 100 40 40,0

2. Strategy and planning 80 44 35,2 2. Strategy and planning 80 34 27,2

3. People 90 50 45,0 3. People 90 24 21,6

4. Partnerships and resources 90 25 22,5 4. Partnerships and resources 90 40 36,0

5. Processes 140 50 70,0 5. Processes 140 27 37,8

6. Customer results 200 25 50,0 6. Customer results 200 23 46,0

7. People results 90 38 34,2 7. People results 90 42 37,8

8. Society results 60 50 30,0 8. Society results 60 11 6,6

9. Key results 150 50 75,0 9. Key results 150 30 45,0

Total 411,9 Total 298,0

Table 17. EFQM assessment results for November 2013 and November 2014

The diagrams in figure 7 give a graphical overview of the tabulated EFQM results. The first

diagram summarises the changes in the tabulated EFQM results by looking at enablers, results

and totals. The second diagram presents the EFQM results within the context of statistical

process control (SPC) where the control parameters are calculated from EFQM results from a

previous study where the action researcher and his team were assessing their action research

process for changing other aspects of NTAX (Øgland, 2013).

The column diagram visualises negative changes in the outcome of EFQM total assessments,

EFQM results assessments and EFQM enabler assessments. What the SPC diagram says is

that the recently observed EFQM totals could both be seen as results of random variations if

the actual EFQM total had been 300 and the use of the EFQM method had been comparable

to how it was used in the period 2000 to 2005. The EFQM statistics confirm what was stated

in the methods section about the difficulty in obtaining validity and reliability when using the

EFQM instrument.

0

50

100

150

200

250

300

350

400

450

Enablers Results Total

2013 2014

0

100

200

300

400

500

600

700

2000

2002

2004

2006

2008

2010

2012

2014

EFQM total AVG = 284.7

UCL = 602.9 LCL = 0.0

Figure 7. Visual comparison of outcome in from EFQM assessment of enablers, results and totals

What the diagrams in figure 7 mean in relation to the research model in figure 4 is that the

study is unable to provide statistical confirmation of hypothesis 1 and hypothesis 2 claiming

the EFQM having an impact on efficiency and effectiveness. On the other hand, when

looking at the qualitative data there is a clear indication of the EFQM model having had an

impact on efficiency, effectiveness and increased understanding on how to implement double-


and single-loop learning. In other words, EFQM measurements are useful but they have to be

handled with caution.

In summary, the EFQM model makes an important contribution at the final stage when

learning is supposed to be specified through a process of reflection as the outcome of the

EFQM model is data that are useful for structuring the process and outcome of the reflection.

Even though the validity and reliability of EFQM self-assessment results may be put into

questioning, especially at early stages when the EFQM model is being used as a general

health check before diagnosing and deciding on treatments, the EFQM outcomes are

nevertheless useful as rough reference points. As the 2014 outcomes are different from the

2013 outcomes, the 2014 diagnosis and action plan for 2015 are expected to be different from

what has been the case for the 2013-14 cycle.

5. Discussion

Although no previous studies have been found to deal specifically with the use of the EFQM

model for improving action research, there have been studies on the use of EFQM for

improving research and development (R&D) in industry and studies on the use of EFQM for

improving university administration.

5.1 The effect of EFQM on single-loop learning

In the NTAX study there is an illustration of single-loop learning in the case of trying to

improve customer satisfaction (criterion 6) by getting external financial funding (sub-criterion

4a). Lack of financial support was an error and writing an application for research grant was

an attempt to correct the error. In the NTAX case the corrective action did not eliminate the

error. The fact that the research application was rejected could be seen as the introduction of

a new error. A possible corrective action for dealing with this would be to learn how to write

better research applications.

More importantly, however, TQM assessment models like the EFQM model function as tools

for single-loop learning in the sense that they force the user to think about the organisation

through a certain list of criteria, sub-criteria, evaluation methods and a language that talks

about issues like suppliers, processes, products, customers and stakeholders. The NTAX

study illustrates some of the challenges in using this language for describing action research.

Endres (1997, pp. 127-129) describes similar experiences at the IBM Watson Research Centre

where there were difficulties in understanding how to interpret each of the criteria and sub-

criteria from the generic TQM model to fit with the context of managing research. In

particular he mentions the customer satisfaction criterion and discusses some of the

challenges in translating the idea of customer satisfaction from the context of manufacturing

into the context of research and development. Despite the challenges, however, his

conclusion is that TQM assessments are important and useful for managing how to

continually improve R&D.

The criterion related to customer results is also specifically dealt with in the NTAX case. As

pointed out by Kock et al (1999), action research serves two masters in the sense of having

customers in industry wanting change and customers in academia wanting knowledge. As the

NTAX study was designed as doing action research in one’s own organisation, it was possible

to use NTAX salary as an indication of NTAX customer satisfaction. In the case of

measuring customer satisfaction among conference reviewers, journal editors and the


scholarly community in general, indicators like peer review feedback, citations and h-indexes

were used.

Another issue of concern at the level of single-loop learning is the design of action strategy.

In the NTAX case the strategy consists of using the 80/20 principle in a simplistic manner for

selecting a couple of issues from the initial EFQM assessment to address. The manner of how

these issues are addressed is based on the causal claims of the EFQM model. In the case of

developing a strategy for dealing with customer results, the EFQM model provides a list of

ten sub-criteria from all of the five enabler criteria. The 80/20 principle was also used for

identifying a couple of sub-criteria on the list, and strategy details were consequently

developed. The way of aligning the EFQM assessments and 80/20 analysis with the cyclic

CAR process is not identical to the RADAR logic defined as part of the EFQM method, but it

is similar in the way the RADAR logic follows along similar steps as those used in the five

steps of action research.

In his account of how TQM assessments were used at the Eastman Chemical Company

(ECC), Endres (1997, pp. 131-137) explain how the strategy was to align the internal

assessment process with the guidelines developed by the National Institute of Standards and

Technology (NIST) for selecting annual Baldrige Award winners. How the ECC

implementation method compares with the NIST guidelines is shown through cross-tables.

The ECC made no use of internal awards but made strong use of improvement focus. The

method used in the NTAX study was similar to the ECC method in this sense, but rather than

being formulated as a twelve step method it was formulated through the use of the five step

action research wheel.

Both the NTAX study and related research seem to confirm that a TQM assessment model

like the EFQM model is useful for stimulating single-loop learning in the sense of having a

public sector research unit adapting to the ideals of NPM. What this means in practice is that

the process of doing and presenting research becomes more similar to the process of dealing

with commodities. The focus changes from the social scientist as an intellectual expressing

himself through socially engaged writing to the image of the scientist as a business-oriented

person trying to optimise his return on investment by strategically researching and publishing

within the context of a scholarly market.

5.2 The effect of EFQM on double-loop learning

In addition to detecting and correcting errors through use of the EFQM model, there are also

recordings of situations in the NTAX study when existing policies and objectives have been

questioned and changed. An example of such a change was the change in method of

measuring customer results and key performance results. By changing the measurement

methods, the assessment of the results changed.

In comparing the use of the EFQM model for improving NTAX action research with the

findings from a study conducted at Corning (Endres, 1997, pp. 130-131), there are similarities

in terms of how the TQM model helps getting an external perspective on the R&D process.

Endres specifically mentions the importance of how the TQM model focuses on customer

satisfaction as a way of making quality management less introspective. A similar effect can

be seen in the NTAX study as the priority of the customer satisfaction criterion makes the

action planning focus on the research-client agreement (RCA). The priority of customer

satisfaction also makes the action planning focus on how to improve action research by

thinking about target audiences for publication of research and target audiences within NTAX


who would be interested in developing an audit system for the quality management system at

SITS. The EFQM model has an automatic effect on double-loop learning in the sense of

being forced to rethink the governing variables in order to optimise action strategies.

As discussed in the case of single-loop learning, however, the EFQM assessment model

makes the organisation adapt to the logic of NPM. The EFQM model stimulates double-loop

learning within the context of NPM, but does not challenge whether NPM is the best ideology

for an organisational unit doing action research. Greenwood and Levin (1998) have argued

that action research as used in quality of life studies in the US betrayed the ideology of the

action research it built upon from studies in the UK, Scandinavia and Australia. Using the

EFQM model for commodifying research could be seen as a similar betrayal of how action

research has been presented as means of researching how TQM can be used for emancipating

individuals and organisational units from managerial and economic oppression (Flood, 1993).

On the other hand, when the EFQM model enforces mercantile language for thinking about

change, the challenge may not necessarily be the language itself but whether the EFQM

model can be seen as consistent with a theory of economics that fits with a critical (post-

Marxist) perspective or not. From a theoretical perspective, however, this challenge has

already been solved by Elster (1982) in his suggestion that Marxist economic theory should

be expressed through the use of game theory.

The effect of EFQM on double-loop learning is that it makes the action researcher question

the relationship between the governing variables, the action strategy and the consequences

and may feel inclined to change the governing variables if the action strategy is not to produce

the right kind of consequences. In order for this to work, however, it is necessary to make use

of game theoretic models that can encompass different types of political interpretations of

economic reality, depending on what the action researcher is trying to achieve. In the NTAX

study this was mentioned in the context of diagnosing the NTAX situation by use of game

theory and the CAR situation by use of control theory. Control theory was used for

improving CAR performance which would then be used for making political impact in the

NTAX situation. Not too much was said about the political situation, however, but this has

been described in more detail in a different theoretical context (Øgland, 2014).

5.3 The effect of EFQM on deutero-learning

In addition to single-loop and double-loop learning, the NTAX study also illustrates learning

about how to facilitate single-loop and double-loop learning. As mentioned at the end of the

previous sub-section, a key tool for facilitating learning is the use of game models. In the

NTAX case it was shown that the EFQM model would more or less automatically generate

single-loop learning in terms of changing practice within the action research unit to make it

comply with the ideals of NPM, and the way the EFQM model provides a dialogue between

enabler criteria and results criteria causes reflection on whether procedures, policies and

governing variables are making the action strategies effective. EFQM assessments can be

used for improving effectiveness and efficiency, but in order to do so it is necessary to

understand whether issues like social justice and environmental sustainability are factors that

should be taken into consideration.

As pointed out by Brunsson and Jacobsson (2000), there is a wide tendency for organisations

to talk about TQM without doing it. Sometimes such organisations are not even aware of the

differences between what they think they are doing and what they actually do. Although one

might expect that EFQM assessments would be helpful for documenting and eliminating

gaps, Brunsson and Jacobsson argue that the problem is not the tools and methods of TQM


itself but the games organisations get enrolled in. On the one hand they may be given

directions to comply with TQM or NPM schemes, but a public sector organisation being

given budgets based on how much the spent last year may not necessarily want to optimise

their cost-efficiency with the consequence of having next year’s budget dramatically reduced.

In the NTAX study there is no explicit game model representing the relationship between the

action researcher, the organisation where research is taking place and the scholarly

community. In more informal terms, however, the game can be characterised by the action

researcher wanting to establish a win-win between NTAX and UiO by convincing both

organisations that there is much to be gained in the long term by collaborating through a

process of doing action research. The aim of the game is to convince both NTAX and UiO

that there is significantly more to gain by collaborating than by burning bridges and going

separate ways. It is this game that guides the way the EFQM method is being interpreted and

used as means for controlling and improving action research.

In terms of related research, Douglas et al (2007) have conducted four case studies on how to

implement EFQM in UK universities, and find that the academic culture has a good fit with

EFQM model and its underpinning concepts. What this assumingly means is that the change

agents have been able to model the game of university administration in a successful way that

allows the EFQM model to take advantage of this model and thus aid performance

improvements in a way that feels natural to the organisation as a whole.

6. Conclusion

6.1 Contributions to theory and practice

This paper has been concerned with the use of the EFQM model for controlling and

improving the action research process. When investigating this approach in practice, by

considering a case of using canonical action research (CAR) for implementing New Public

Management (NPM) within the Norwegian tax administration (NTAX), the EFQM approach

had effects on action research performance on three different levels.

The first effect of EFQM on CAR was observed in terms of what is sometimes referred to as

single-loop learning (Argyris & Schön, 1978). Regardless of how the political and practical

aims of the action research were defined, in order to get good results from the EFQM

assessments it was necessary to view the outcome of the action research process through a

balanced scorecard perspective that would include looking not only at results in terms of

scientific publications but also at customer satisfaction (both client and scholarly community),

internal satisfaction within the action research team, and impacts on society. Furthermore, for

each of these four domains it was necessary to make use of both perception measurements

and performance measurements.

The second effect of EFQM on CAR was observed in terms of double-loop learning, meaning

that the identification and correction or errors done at the level of single-loop learning

resulted in reflections on whether the governing variables of the action research could be

improved. Part of the reason for double-loop learning had to do with the nature of the EFQM

model itself in terms of EFQM theory provides links between TQM enablers and results.

Another part of the reason for double-loop learning had to do with how this relationship

between enablers and results in the EFQM model was integrated in the action planning stage

of CAR. In other words, the use of the EFQM model provides a natural framework for

reflecting both on the results of CAR and what enables the results.


The third effect of EFQM on CAR is that it provides a framework for learning how to

implement action research as single-loop and double-loop learning. Among double-loop

learning scholars (e.g. Argyris & Schön, 1978), this type of learning is sometimes referred to

as deutero-learning. The EFQM model enhances what Schön (1983) refers to as reflection-

on-action in the sense that it proposes a set of 9 criteria and 32 sub-criteria that are relevant

for all kinds of organisations that want to improve performance from a TQM perspective.

While previous research has showed that this kind of approach works fine in university

environments and in industrial R&D units, this particular study suggests that the EFQM

model can also provide single-loop, double-loop and deutero-learning for the action research

team as an organisation in itself.

In terms of related research, action researchers with an interpretivist orientation have

suggested ways of improving action research by focusing on ethics and standards (Boog et al,

2008), and action researchers with a positivist orientation have suggested ways of improving

action research by using the cyclic structure of the approach to increase generality, increase

control over environments and reduce the role of the individual researcher (Kock et al, 1997).

In using the EFQM model for improving action research, however, neither an interpretive nor

a positivist approach has been used. The improvement strategy has been aligned with the idea

of using design science for articulating action research (Simon, 1996; Järvinen, 2007; Øgland,

2009) in terms of using concepts like systems, feedback, control, decisions and games.

Rather than improving the diagnosis as a goal in itself (interpretivist concern) or improving

the validity of how the treatment is being tested as a goal in itself (positivist concern), the

design science approach is concerned with theorising the context and design of effective and

efficient actions from the practitioner perspective. In other words, when the action researcher

uses the EFQM model for improving his own performance, he is acting in alignment with the

type of action research sometimes referred to as reflective practice (Schön, 1983; McNiff,

2000; McNiff & Whitehead, 2006). The study shows that this is a useful perspective when

trying to create organisational learning and change in a case like trying to establish a NPM

audit system in a politically complex organisation.

6.2 Limitations and directions for further research

The study has several limitations that may suggest further possibilities for empirical research.

First, the effects of EFQM on CAR are studied in the context of an action research project that

is still at a definition stage despite covering almost a year of action. There is need for

investigating the effect of EFQM on CAR in cases where the action research has gone beyond

the establishing of a researcher-client agreement (RCA) and is engaging more directly with

the client organisation.

Second, the EFQM assessments in the study were done in a simplistic manner that was

sufficient from the viewpoint of the maturity of the CAR process. However, for the EFQM

model to be a useful tool for control and improvement it is necessary to establish a

trustworthy baseline score that can be used as basis for evaluating the impact of the action

research interventions being defined and implemented. Third, an obvious benefit in using the

EFQM for assessing CAR projects is that it makes it possible to compare the TQM

performance of one CAR project or process against another. This, however, requires that

similar studies are being carried out in other organisations by other action researchers.


Acknowledgement

The researcher wants to thank Prof. Jens Kaasbøll for interesting discussions that have lead

towards the development of this paper.

References

Aldridge, J., & Derrington, A. M. (2012). The Research Funding Toolkit: How to Plan and

Write Successful Grant Applications. London: SAGE.

Argyris, C. (1994). Good communication that blocks learning. Harvard Business

Review, 72(4), 77-85.

Argyris, C., Putnam, R., & Smith, D. M. (1985). Action science. San Francisco: Jossey-Bass.

Argyris, C., & Schön, D. (1978). Organizational learning: A theory of action

perspective. Massachusetts: Addison-Wesley Publishing Company.

Berry, D. (2010). Gaining funding for research: A guide for academics and institutions. New

York: McGraw-Hill International.

Boog, B., Preece, J., Slater, M., & Zeelen, J. (2008). Towards quality improvement of action

research: Developing ethics and standards. Rotterdam: Sense publishers.

British Quality Foundation (2000). The model in practice: Using the EFQM excellence model

to deliver continuous improvement. Exeter, Devon: British Quality Foundation.

Brunsson, N., & Jacobsson, B. (2000). A world of standards. Oxford: Oxford University

Press.

Clark, P. A. (1972). Action research and organizational change. London: Harper & Row.

Cole, R., Puaro, S., Rossi, M. & Sein, M. (2005). Being Proactive: Where Action Research

Meets Design Research. International Conference on Information Systems (ICIS), 2005.

Davies, J., Douglas, A., & Douglas, J. (2007). The effect of academic culture on the

implementation of the EFQM Excellence Model in UK universities. Quality Assurance in

Education, 15(4), 382-401.

Davison, R., Martinsons, M.G & Kock, N. (2004). Principles of Canonical Action Research,

Information Systems Journal, 14(1), 65-86.

DNV (2005). Kvalitetsarbeid i IT-funksjonen, Vurdering av systemet for kvalitetsstyring i IT-

funksjonen i Skattedirektoratet, Rapport unntatt offentlighet, 03.10.2005, Det Norske

Veritas: Høvik.

Elster, J. (1982). The case for methodological individualism. Theory and society, 11(4), 453-

482.

Endres A. C. (1992). Results and Conclusions from Applying TQM to Research. ASQC

Quality Congress Proceedings, Milwaukee.

Endres, A. C. (1997). Improving R&D Performance The Juran Way. New York: John Wiley

& Sons.

Fisher, R. & Ury, W. L. (1981). Getting to yes: Negotiating agreement without giving in.

London: Penguin.

Flood, R.L. (1993). Beyond TQM. Chichester: Wiley.

Fukuyama, F. (2013). What is governance? Governance, 26(3), 347-368.

Gartner (2010). Root cause analysis - en rapport for Skatteetaten. Version 1.1, February 4th

2010, internal publication, Skatteetaten.

Gold, M. (1999). The Complete Social Scientist: A Kurt Lewin Reader, Washington:

American Psychological Association.

Greenwood, D. J., & Levin, M. (1998). Introduction to action research: Social research for

social change. London: Sage.

Hides, M. T., Davies, J., & Jackson, S. (2004). Implementation of EFQM excellence model

self-assessment in the UK higher education sector–lessons learned from other sectors. The

TQM Magazine, 16(3), 194-201.


Hood, C. (1991). A public management for all seasons?. Public administration, 69(1), 3-19.

Huff, A. S. (1999). Writing for scholarly publication. London: Sage.

Järvinen, P. (2007). Action research is similar to design science. Quality & Quantity, 41(1),

37-54.

Koch, R. (1998). The 80/20 principle: the secret to achieving more with less. London:

Nicholas Brealey Publishing.

Kock, N., Avison, D., Baskerville, R., Myers, M., & Wood-Harper, T. (1999). IS action

research: can we serve two masters? In Processings of the International Conference on

Information Systems (pp. 582-585).

Kock Jr, N. F., McQueen, R. J., & Scott, J. L. (1997). Can action research be made more

rigorous in a positivist sense? The contribution of an iterative approach. Journal of

Systems and Information Technology, 1(1), 1-23.

Kontostavlaki, D., Kontou, E., Diamantidis, F., & Dimopoulus-Secondary, V. (2010). Self-

assessment and improvement plans through action research, according to the EFQM

model of excellence. Action Research in Diverse Contexts: Contemporary Challenges, 31.

Krick, E.V. (1962). Methods Engineering: Design and Measurement of Work Methods.

New York: John Wiley & Sons.

Laudel, G. (2006). The art of getting funded: how scientists adapt to their funding

conditions. Science and Public Policy, 33(7), 489-504.

Lilford, R., Warren, R., & Braunholtz, D. (2003). Action research: a way of researching or a

way of managing?. Journal of health services research & policy, 8(2), 100-104.

Lyytinen, K., & Robey, D. (1999). Learning failure in information systems

development. Information Systems Journal, 9(2), 85-101.

McKay, J., & Marshall, P. (2001). The dual imperatives of action research. Information

Technology & People, 14(1), 46-59.

McNiff, J. (2000). Action Research in Organisations. London: Routledge.

McNiff, J., & Whitehead, J. (2006). All you need to know about action research. London:

Sage publications.

Mintzberg, H. (1983). Structures in Fives: Designing Effective Organizations. Upper Saddle

River, New Jersey: Prentice-Hall.

Mintzberg, H. (1996). Managing government, governing management. Harvard Business

Review, 74(3), 75-83.

Novak, J. D. (1984). Learning how to learn. Massachusetts: Cambridge University Press.

NTAX (2005). Kvalitetsstyring av IT-funksjonen i Skatteetaten, Kvalitetsfunksjonens

egenvurdering 2004, SKD 2004-106, January 25th 2005, internal report,

Skattedirektoratet, Helsfyr, Oslo.

Oakland, J. S. (1999). Total Organizational Excellence: Achieving world-class performance.

Oxford: Butterworth-Heinemann.

Oakland, J. S. (2003). Total quality management: text with cases. London: Routledge.

Øgland, P. (2007). Improving research methodology as a part of doing software process

improvement. In Proceedings of the 30th Information Systems Research Seminar in

Scandinavia IRIS (pp. 1-18).

Øgland, P. (2009). Action Research and Design Science Research - More similar than

dissimilar, In Artikkelsamling for den 16nde norske konferanse for organisasjoners bruk

av informasjonsteknologi (NOKOBIT 2009), Trondheim, Norway, pp. 171-184.

Øgland, P. (2013). Mechanism design for total quality management: Using the bootstrap

algorithm for changing the control game (Doctoral dissertation, University of Oslo).

Øgland, P. (2014). The principle of Researcher-Client Agreement in Canonical Action

Research. Den 11te årlige konferanse i Nettverk for organisasjonsforskning i Norge


(NEON-dagene 2014): "Bærekraftig organisering – nye handlingsrom?", 25-27

November 2008, Stavanger, Norway.

Prybutok, V. R., & Ramasesh, R. (2005). An action-research based instrument for monitoring

continuous quality improvement. European Journal of Operational Research, 166(2),

293-309.

Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New

York: Basic books.

Simon, H. A. (1996). The sciences of the artificial. Third Edition. Cambridge, Massachusetts:

MIT press.

Simonsen, J. (2009). A Concern for Engaged Scholarship: The challenges for action research

projects. Scandinavian Journal of Information Systems, 21(2), 111-128.

Statskonsult (2002). Organisering av IT-funksjonen i skatteetaten, Rapport 2002: 13. Oslo:

Statskonsult.

Sternberg, R. J. (Ed.). (2013). Writing Successful Grant Proposals from the Top Down and

Bottom Up. London: SAGE Publications.

Wroldsen, K.O. (2008). Skatteetaten – 50 år med IKT og omstilling. In: Jansen, A. &

Schartum, D.W. Elektronisk forvaltning på norsk: Statlig og kommunal bruk av IKT.

Bergen: Fagbokforlaget (pp. 119-144).

the effect of efqm self-assessments on action …...as the purpose of the efqm model is to help...

Documents