nice model, but what is the evidence?

Nice model, but what is the evidence?

EHMA Annual Meeting, Porto, June 15th, 2016

Evidence Based Practice, basic principles and practical examples

Postgraduate Course

Our Fellows

CEBMa: what we do

Promote (seminars, papers, blogs, tweets)

Educate (universities & business schools)

Train & coach (companies > projects)

Support / REAs (companies)

Support / 2nd opinion (BS detector)

15 min Management models

25 min EBP: What is it and why do you need it?

20 min A practical example

… min Discussion

Our mission for today

To disturb

To reassure

Dangers of believing things that aren’t true

It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so. (Mark Twain)

The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge. (Stephen Hawking)

Some examples

What’s happened to average job tenure in the UK and US over the past 10 years?

Up?

Down?

Stayed the same?

Some examples: Job satisfaction

What’s happened to job satisfaction in the US over the past 10 years?

Up?

Down?

Stayed same?

Some examples: VUCA

Is the world moving ever faster – more VUCA (volatility, uncertainty, complexity and ambiguity)?

Faster?

Slower?

Same?

The idea that time is speeding up is clearly popular. It is also plausible. There is just one problem. It is very hard to prove that it is actually happening.

Hard evidence of a great acceleration is hard to come by. The Economist has considered a variety of measures by which the speed of business in America can be quantified. A few do show some acceleration. But a lot do not.

UK Patents

Some examples: Generational differences

Are there generational differences in work attitudes?

Big differences?

Small differences?

No differences?

So what?

There are many taken-for-granted assumptions and ‘truths’ in management that turn out to be wrong or at least overstated

It’s important to examine these assumptions because if they are wrong will lead to poor decisions

Possibly also in the case of Management Models

Nice pictures

Model

Taken-for-granted assumptions?

Fad? Evidence?

The question is …

Model

Most of these models are developed within the domain of management

The problem with managers

They love fads!

38

Scientific Management/Taylorism

Business Process Reengineering

Total Quality Management

Learning Organizations

Knowledge Management

The nearly forgotten fads

39

40

Talent management

Emotional intelligence

Employee engagement

Lean / six sigma / theory of constraints

Autonomous teams (it’s back!)

Agile

The fads that haven’t been forgotten (yet)

41

Fads seem to be attractive, compelling and irresistible

Promise to deliver a lot and fast

Appear simple

Will make everything alright and help contain

anxieties around intractable problems

Help user feel effective and cutting edge

Bits of some fads may work in some context

43

44

1. What tells you a model is a fad?

2. Can you give an example in healthcare management?

Discuss with your neighbour (2 min)

Discussion

60595857565453525150494847464544434241403938373635343332313029282726252423222120191817161514131211109876543210

1. Simple, straightforward: Easy to communicate, comprehend, and reduce to a small number of factors or dimensions.

2. Promising results: Fad auteurs are confidently didactic. There is no false humility or hedging.

3. Universal: Solutions for everyone everywhere.

4. Step-down capability: Can be implemented in ritualistic and superficial ways.

How to spot a management fad

5. In tune with zeitgeist: Resonate with trends or perceived business problems of the day.

6. Novel, not radical: Novelty is not so much new discovery as a rediscovery and repackaging of older ideas.

7. Lively, entertaining: Articulate, bold, memorable and upbeat. Filled with labels, buzzwords, lists, acronyms and fun anecdotes > your system 1 loves it!

8. Legitimacy via gurus and star examples: Supported by stories of excellent companies and the status of gurus, not by solid evidence.

How to spot a management fad

Model

So, we need a meta-model to check

whether the model is evidence based

Evidence based practice:What is it?

Evidence based medicine?

Never heard off

Heard off

Heard off and can explain

Evidence based management?

Never heard off

Heard off

Heard off and can explain

Evidence-based practiceCentral Premise:

Decisions should be based on the ‘best available evidence‘.

Evidence?

information, facts or data supporting (or contradicting) a claim, assumption or hypothesis

Evidence?

outcome of scientific research, organizational facts & figures, benchmarking, best practices,

personal experience

All managers and leaders base their decisions on ‘evidence’

But…many managers pay little or no attention to

the quality of the evidence they base their decisions on

and use too few sources of evidence

Trust me, 20 years of management experience

Sources of evidence

problem solution

Practitionersprofessional expertise

Organization internal data

Stakeholdersvalues and concerns

Scientific literature empirical studies

AskAcquire

AppraiseAggregate

ApplyAssess

Professional experience and

judgment

Multiple sources?

Multiple sources?

Organizational data, facts and figures

Multiple sources?

Organizational data, facts and figures

Multiple sources?

Scientific research

outcomes

Evidence based practice:Where does it come from?

McMaster University Medical School, Canada

Medicine: Founding fathers

David Sackett Gordon Guyatt

How it all started

1. Ask: translate a practical issue into an answerable question

2. Acquire: systematically search for and retrieve the evidence

3. Appraise: critically judge the trustworthiness of the evidence

4. Apply: incorporate the evidence into the decision-making process

5. Assess: evaluate the outcome of the decision taken

5 steps of EBmed

Evidence-Based Practice

1991Medicine

1998Education

2000Social care, public policy

Nursing, Criminal justice,

Policing, Architecture, Conservation

2010Management

Evidence-Based Practice

Evidence-Based Practice

Evidence-based … whatever =

the use of evidence from multiple sources to increase the likelihood of a

favourable outcome

Focus on the decision making processThink in terms of probability

Evidence-Based Decision-Making Why do we need it?

It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so. (Mark Twain)

The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge. (Stephen Hawking)

Advice: lie babies down to sleep on their belly(unanimous support through to the 1990s)

Example: medicine

Nr of cot deaths (Holland)

Example: economy / finance

Nov 15, 2005"With respect to their safety, derivatives, for the most part, are traded among very sophisticated financial institutions and individuals who have considerable incentive to understand them and to use them properly.”

Jan 10, 2008”The Federal Reserve is not currently forecasting a recession”

Collateralized Debt Obligations > AAA

0.12% (about 1 chance in 850) default in 5 years

Example: economy / finance

Forecasted Actual

Forecasted and actual 5-year default rates for AAA-rated CDO tranches

0.12%

28%

5 Trillion dollars in pension money, real estate value, savings and bonds disappeared

8 million people lost their job

6 million people lost their houses

Example: economy / finance

(and that was only in the USA)

Scared straight

Example: policy / prevention

Example: HR management

1. Incompetent people benefit more from feedback than highly competent people.

2. Task conflict improves work group performance while relational conflict harms it.

3. Encouraging employees to participate in decision making is more effective for improving organizational performance than setting performance goals.

Likely or unlikely?

ALL NOT LIKELY !

How evidence-based are HR managers?

959 (US) + 626 (Dutch) HR professionals 35 statements, based on an extensive body of

evidence true / false / uncertain

HR Professionals' beliefs about effective human resource practices: correspondence between research and practice, (Rynes et al, 2002, Sanders et al 2008)

Outcome: not better than random chance

Example: management models

George BuckleyJames McNerney

So where does the evidence-based stuff come in?

Relying on only 1 source: bad idea!

problem solution

Practitionersprofessional expertise

Organization internal data

Stakeholdersvalues and concerns

Scientific literature empirical studies

Discuss with your neighbor (1 min)

Over a 5 year period,

why is an orthopedic surgeon's experience, as a rule, more trustworthy than a change manager’s experience?

60595857565453525150494847464544434241403938373635343332313029282726252423222120191817161514131211109876543210

Developing expertise

1. A sufficiently regular, predictable environment

2. Numerous opportunities to practice

3. Receive accurate (objective) feedback

The management domain is not highly favorable to expertise!

Bounded rationality

How your brain works

System 1 Fast Intuitive, associative heuristics & biases emotional

System 2

Lazy Slow Deliberate Rational

dominant

System 1: short cuts

System 1 or system 2?

10 seconds

System 1 or system 2?

A bat and a ball cost $1.10 in total.

The bat costs $1 more than the ball

How much does the ball cost?

109876543210

System 1: necessary to survive

95%

Pattern recognition Overconfidence bias Halo effect False consensus effect Group think Self serving attribution bias Sunk cost fallacy Cognitive dissonance reduction

System 1: prone to cognitive errors

Confirmation bias Authority bias Small numbers fallacy In-group bias Recall bias Anchoring bias Availability bias

When it is your job to make decisions,

you need to know how your brain works

when making decisions!

We are predisposed to see order, pattern and causal relations in the world.

Patternicity: The tendency to find meaningful patterns in both meaningful and meaningless noise.

pattern recognition

We are pattern seeking primates: association learning

pattern recognition

Points of impact of V-1 bombs in London

Points of impact of V-1 bombs in London

Pattern recognition

Pattern recognition

A Type I error or a false positive, is believing a pattern is real when it is not (finding a non existent pattern)

A Type II error or a false negative, is not believing a pattern is real when it is (not recognizing a real pattern)

Dr. Michael Shermer (Director of the Skeptics Society)

Pattern recognition

A Type I error or a false positive: believe that the rustle in the grass is a dangerous predator when it is just the wind (low cost)

Pattern recognition

A Type II error or a false negative: believe that the rustle in the grass is just the wind when it is a dangerous predator (high cost)

Pattern recognition

A Type I error or a false positive: believe that the rustle in the grass is a dangerous predator when it is just the wind (low cost)

A Type II error or a false negative: believe that the rustle in the grass is just the wind when it is a dangerous predator (high cost)

DEFAULT

Pattern recognition

Doctors and managers hold many erroneous beliefs,

not because they are ignorant or stupid, but because

they seem to be the most sensible conclusion

consistent with their own professional experience!

“I’ve been studying judgment for 45 years, and I’m no better than when I started. I make extreme predictions. I’m over-

confident. I fall for every one of the biases.”

Practitionersprofessional expertise

Organization internal data

Stakeholdersvalues and concerns

Scientific literature empirical studies

Four sources of evidence (not only 1)

The performance of knowledge workers

A Practical Example

550 beds

3300 employees

210 medical specialists

225,000 admissions

Top Clinical & Teaching hospital

Organization

The model

2015: 7.2 2016: 6.3

A happy & engaged employee is a productive employee

Fundamental assumption

Let’s have a look at the evidence

Professional experience and

judgment

Organizational data, facts and figures

Stakeholders’ values and concerns

Scientific research

outcomes

AskAcquire

AppraiseApply

Assess

problem solution

Productivity: mean observed r = .15 (𝞀 = .25)

Correlation with job satisfaction r = .91

… The effect of job attitudes (e.g. satisfaction, commitment)

on performance was weak (β = .06) ….

… The relationship was almost eliminated after controlling for

personality traits …. and self esteem.

GREAT! NOW WHAT?

Evidence-based approach, step 1: ASK

problem solution

Practitionersprofessional expertise

Organization internal data

Stakeholdersvalues and concerns

Scientific literature empirical studies

AskAcquire

AppraiseAggregate

ApplyAssess

Population? Knowledge workers

Whether nurses, lawyers, engineers, managers, or staff members, nowadays most workers in organizations are highly dependent on information and communication technology and are involved in

work that involves a high level of cognitive activity.

Question

“Which of the factors that are related to the

performance of knowledge workers are most

widely studied and what is known of their

effect?”What do you think?

I Don’t Know(but I know how to find out)

The 3 hardest words in management

Step 2: ACQUIRE

Search for the best available scientific evidence

ABI, BSP, PsycINFO

Scholarly journals, peer reviewed

1980 – 2013

English

performance, productivity, knowledge work*

ACQUIRE

step 3: APPRAISE & AGGREGATE

Effect size?

Most studied & largest effect size

1. Social cohesion .5 / .7

2. Perceived supervisory support .5

3. Information sharing / TM.5

4. Vision / goal clarity.5

5. Trust.3 / .6

step 3b: CROSS VALIDATE – multiple sources

Step 4: APPLY

Three examples

social cohesion supervisory support

information sharing

Social cohesion

Social cohesion

… a shared liking or team attraction that includes bonds of friendship, caring,

closeness, and enjoyment of each other’s company.

Social cohesion

Measuring social cohesion

Perceived supervisory support

…how employees feel the supervisor helps them in times of need, praises

them for a job well done or recognizes them for extra effort.

Perceived supervisory support

Perceived supervisory support

Measuring perc. sup. support

Information sharing

Information sharing?

…refers to how teams pool and access their knowledge and expertise – which positively

affects decision making and team processes. This has led to the idea of a team ‘Transactive

Memory System’ (TMS), which can be thought of as a collective memory in a collective mind - enabling a team to think and act together

Information sharing

Measuring information sharing

Step 5: ASSESS the outcome

The departments with the lowest performance scored under average on most factors

Reactions

Who knew?

So, when a new model crops up