cihi’s performance measurement framework journey: the ... › wp-content › uploads › 2017 ›...

@NHLC2018 #NHLC2018

CIHI’s Performance Measurement Framework Journey: the challenges and rewards

Presenters: Sandra Mitchell, Manger, Governance and Strategy, CIHI Karen Weir, Program Lead, Governance and Strategy, CIHI

2 2

CIHI’s PMF Journey

• PMF Overview

• PMF Indicator Development

• Reporting

• Challenges and Rewards

• Next steps in the Journey

Achieve Results: Learning Objective: Learn how an effective performance measurement framework can facilitate accountability and lead to change.

3 3

CIHI

• Created in 1994

• Independent, not-for-profit organization that provides essential information on Canada's health systems

• Led by a 16-person Board of Directors, with representation from across the country

4

CIHI’s PMF Journey

Strategic Plan Performance

Measurement Framework

Logic Model Indicators Targets Reporting

5 5

  PMF Overview

6

CIHI’s Strategic Plan 2016 to 2021

7 7

Purpose of the Performance Measurement Framework

• Guide the measurement of CIHI’s progress in achieving its 2016-2021 Strategic Plan

• Tool provides senior leadership with concrete data and information to:

‒ make sound decisions

‒ continuously improve performance

8 8

Key Features

• Aligned with Strategic Plan

• Demonstrating results – impact and outcomes

• Measuring actionable elements

• Based on accurate and reliable performance data

• Consistency with Treasury Board / Health Canada requirements

9 9

Process to develop PMF and Logic Model • Steering Committee (January – March 2016)

‒ Executive Team, Directors, Governance and Strategy

• PMF was developed through a series of 8 working sessions

‒ CIHI profile

‒ Logic model

‒ Identification of the associated PMF indicators

‒ measuring achievement against the Annual Plan

10 10

CIHI’s Logic Model

11

Overview

12 12

Activity Stream 1: Standards and Quality Data   Activity/Box 1: Develop data standards and perform

quality assurance • Obtain consensus on health information standards

• Develop standards in priority areas and support their adoption

• Increase timeliness of data receipt and exchange

  Activity/Box 2: Acquire/collect data and develop/implement data access • Acquire/collect data in priority areas

• Implement data access strategy

• Develop flexible data collection tools streamlining the data collection process

  Output/Box 7: Data and Standards • Enhanced and optimized library of data standards and enhanced data

quality tools and assessments

• Accessible data sources in priority areas, enhanced coverage of existing data holdings

13 13

PMF Indicators

• Developed 19 PMF indicators

‒ 11 PMF indicators reported to the Board • Specific indicators developed for each output and outcome

‒ Output indicators: largely “counts” of completed products

‒ Outcome indicators: measure stakeholder use, access, and behavioural changes as a result of CIHI data and products

14 14

A Few PMF Indicators

• Output

‒ % of analyses released that align with priority population themes

• Immediate Outcomes

‒ Increase in access to CIHI’s public data

• Long-Term Outcomes

‒ Extent to which CIHI has contributed to health system improvements

15 15

  Developing PMF Indicators

16 16

Process to develop PMF Indicator Measures • Working Group established November 2016

• Managers and Directors

‒ Chaired by Governance and Strategy

• For each indicator

‒ Determine scope (products and services)

‒ Define appropriate measures

‒ Identify data sources and frequency of reporting

‒ Identify targets

17 17

Targets

• Target setting occurs during Annual planning process

• Set by Executives

• May cross divisions

• Targets shared with the Board’s Governance and Privacy Sub-Committee

18 18

Reporting • Semi-annual data collection for select indicators

• Annual data collection for all indicators

• Data sources

‒ CIHI administrative data

‒ Analytical plan

‒ External stakeholder satisfaction survey

‒ to support 4 indicators based on stakeholder feedback

‒ Vignettes

19 19

Long-term Outcomes • Indicators:

‒ Extent to which CIHI has contributed to pan-Canadian population health

improvements

‒ Extent to which CIHI has contributed to health system improvements

• Track long-term outcomes on how CIHI’s products and services are contributing to population health and health system improvements

• CIHI follows up with stakeholders to learn how its products and services are used to identify an opportunity for improvement

• Triggers an implementation plan to improve an outcome

• Subsequent product and service reporting demonstrates improvements

20 20

Your Health System (YHS): Helping a Hospital focus on key quality-

of-care initiatives

• YHS: In Depth, In-Hospital Sepsis indicator 2014-2015

• Results for In-Hospital Sepsis indicator different from their expectations, from peer, regional and provincial comparators

• Urinary tract infections (UTIs) were the greatest contributors to the In-Hospital Sepsis results

• Goal: To decrease surgical catheter–associated UTIs from 2.5% to 1.7% by April 2016.

• Action: Implemented an initiative to reduce the number of catheter insertions and the number of catheter days

• Outcomes: UTI rate dropped from 2.5% to 1.5%

21 21

  Challenges and Rewards

22 22

Challenges of Developing the PMF and Logic Model

• Not the natural ‘day job’ of executives or managers

• Discussions with Health Canada resulted in changes along the way

• Took more time than anticipated

23 23

Developing the Indicator Definitions and Indicator Measures

• People who developed the indicators were not at the table to develop the measures

• Managers often had more insight on what could or could not be measured

‒ In some cases this led to the rephrasing and re-scoping of the indicators

• Indicators created based on what needs to be measured

‒ i.e. not based on what data is readily available

24 24

Collecting and Capturing Indicator Data

• Staff responsible for providing the data were not involved in the development of the indicators or the measures

‒ therefore did not understand data requirements or rationale

‒ not the natural ‘day job’ for staff,

‒ concept of Performance Measurement is not intuitive

• Request for data to numerous departments and numerous staff

• Conflicting data between departments for the same measure

• Provision of data particularly arduous as departments had not previously been asked to track data in the format required for the indicators

25 25

Reporting on Indicator Results

• Choosing what to highlight

• Ensuring that the results have been explained appropriately

• Challenge to explain why there are inconsistencies in targets and actuals achieved

26 26

Rewards

• Year one (2016-2017) reported:

‒ 14 out of 19 indicators

‒ 1 long term outcome

‒ 6 out of 13 indicators were meeting or exceeding targets

• Year two (2017-2018) reported:

‒ 18 out of 19 indicators

‒ 3 long term outcomes and tracking many more

‒ 9 out 13 indicators were meeting or exceeding targets

27 27

Take Away

• Different way of thinking and new skill set for the organization

• Developed new indicator measures where there are no pre existing data collection methods in place

• Set up as a project – can’t be done in a short timeframe off the side of people’s desk

• Identification and capture of performance measurement indicator measures is an evolving process

28

Next Steps in the CIHI PMF Journey

Strategic Plan Performance

Measurement Framework

Logic Model Indicators Targets Reporting

cihi.ca cihi.ca @cihi_icis xxxx@cihi.ca @personal twitter handle

31

Performance Indicators - Outputs Area Indicator Frequency

Data and standards # of planned new and major standard enhancements Annual

# of planned data quality assessments completed by CIHI Annual

Increase (%) in coverage of data collection in priority areas * Annual

Analytical tools and methods

% of planned new and enhanced analytical tools completed by CIHI

Annual

% of planned health information infrastructure enhancements completed within the year

Annual

Health information infrastructure

% of planned health information infrastructure enhancements completed within the year

Annual

% of data sources included in integrated e-reporting (IeR) against the 2021 target

Annual

Analytical products % of analyses released that align with priority population themes*

Annual

Knowledge exchange / capacity development

# of capacity development events or activities Annual / semi-annual

32

Performance Indicators – Immediate Outcomes Area Indicator Frequency

Increased access to quality, integrated data

Increase in access to CIHI’s public data* Semi-annual

% improvement in the quality of the data Annual

# of linked data files available through third parties*

Annual

Increased access to analytical tools and products

Level of stakeholder satisfaction with access to and usefulness of tools and products*

Periodically

Increase in total number of users of CIHI’s private online tools/products*

Semi-annual

Increased stakeholder knowledge and capability to use products and services

% of stakeholders (target group) reporting increased knowledge (awareness) of CIHI products and services in their setting*

Periodically

% of stakeholders (target group) reporting increased capability to use CIHI products and services in their setting*

Periodically

33

Performance Indicators – Intermediate and Long-Term Outcomes

Area Indicator Frequency

Intermediate: Increased use of CIHI products and services to support decision making

% of stakeholders who report using evidence from a CIHI knowledge product or service to support decision-making in their setting*

Periodically

Long-term: Contribution to pan-Canadian population health and health system improvements

Extent to which CIHI has contributed to pan-Canadian population health improvements*

Annual

Extent to which CIHI has contributed to health system improvements*

Annual

*Indicates PMF indicators that are reported to the CIH Board of Directors