module 2: using a data dictionary · the pm numerator, or “top part of the fraction” - parts of...

$: Module 2: Using a Data Dictionary · The PM Numerator, or “top part of the fraction” - parts of the whole that meet identified criteria being measured: Number of staff who received$
PA Office of Developmental ProgramsQuality Management Recertification

1

Module 2: Using a Data Dictionary

Hello and welcome to Part 2 of ODP’s QM Recertification Series 2020, Using a Data Dictionary.

1

Objectives for Recertification Part 2

2

• Identify ODP’s definition of Data Dictionary

• Understand benefits of using a Data Dictionary

• Refresh Knowledge about Performance

Measurement

• Describe Data Dictionary content

Upon completion of this training, you will be able to meet the following objectives:

• Identify ODP’s definition of Data Dictionary

• Understand the benefits of using a Data Dictionary

• Refresh you knowledge of Performance Measurement

• Describe Data Dictionary content

2

Definition

3

Quality Management “road map” that keeps users

on track to consistently and accurately collect,

interpret and analyze data

So what is a Data Dictionary?

If you “Google” the term, you will see a number of different definitions, many of which are used by those in the Information Technology field.

In ODP, we use the term differently. We like to think of it as a Quality Management “road map” that keeps users on track to consistently and accurately collect, interpret,and analyze data.

3

Benefits

4

• Records critical data-related information in one

central place

• Uses operational definitions to clarify

understanding of performance measures

• Provides consistency in data collection and

analysis interpretation

Use of a Data Dictionary can provide organizations many benefits, including:

• Records critical data-related information in one central place

• Uses operational definitions to clarify understanding of performance measures:o It is not uncommon for different members on the same team within an

organization to interpret a concept or term differently. When data is involved, this can result in confusion and disagreement about what the data show (the findings) and what actions, based on the data, need to be taken

o Provides CONSISTENCY with data collection and analysis/interpretation

4

Benefits (continued)

5

• Inventories list of performance measures

• Improves trust in data integrity

• Supports/assists in decision-making

A Data Dictionary also:

• Inventories your list of performance measures

o as we mentioned earlier, you can list all of your performance measures and accompanying information on one Data Dictionary

• Improves trust in data integrity

o data integrity is the maintenance of, and accuracy/consistency of data over its entire life-cycle

o use of a Data Dictionary also assures data integrity through consistency in application

• Supports/assists in decision-making

o decisions are better made through use of accurate data rather than reliance on anecdotal information and/or intuition

5


6

• Contributes to a Learning Organization culture

that:

o Supports team-based structure

o Identifies roles and responsibilities

o Promotes accountability

o Saves time

A Data Dictionary contributes to a Learning Organization culture in the workplace that:

o supports a team-based structure

o identifies roles and responsibilities

o promotes accountability for completing tasks in a process

o saves time in the long-run by making sure steps in the data collection and analysis process are done correctly/accurately the first time around (no period)

6


7

• Facilitates transfer of knowledge

• Assures everyone understands the information in

the same way

• Functions as a communication tool for

Leadership

And finally….

• A Data Dictionary facilitates transfer of knowledge when new team members join an organization or current staff transitioning to new responsibilities.

o Every organization experiences staff turnover from time to time. Use of a Data Dictionary ensures a smooth transition process, especially when staff are tasked with important data-related responsibilities.

• Assures everyone understands the information in the same way

• Functions as a communication tool for Leadership; e.g., when discussing the rationale for a QM Plan

Before starting to build a Data Dictionary, let’s review some basics about performance measurement.

7

8

Performance Measures (PM)

Performance measures are the way to measure progress towards achieving goals and objectives.

8

• Structure: Looks at aspects essential for

operation and the ability to provide service

• Process: Assesses the degree to which the

appropriate or expected course of action occurs

• Outcome: Evaluates the degree to which the

expected outcome was met

PMs: Three Basic Types

9

There are three basic types of performance measures:

• Structure Performance Measure - looks at aspects essential for operation and the ability to provide service, such as appropriate number of staff

• Process Performance Measure - assesses the degree to which the appropriate or expected course of action occurs, such as whether a yearly screening test occurs as required

• Outcome Performance Measure - evaluates the degree to which the expected outcome was met. For example, the percentage of individuals whose assessed needs are met or the number of people desiring employment who are working

9

• a simple number, or count

• an equation …

PMs: Count vs. Equation

10

9090

90100

Remember, a performance measure may be a simple number, or count, such as how many individuals attended a community event.

Or, you might instead use an equation to calculate the percentage of individuals who

attended a community event. In this case:

• the numerator, "Top part of the fraction" - parts of the whole that meet

identified criteria being measured - would be the number of individuals who

attended a community event

• the denominator, "Bottom part of the fraction” - the whole being measured -

would be the total number of individuals who potentially could have attended a

community event

10

PMs: What to Keep in Mind

11

• Reflect organizational goals

• Be key to its success

• Be measurable

• Tell you what you need to know

Performance measures you select for your organization must:

• Reflect organizational goalso helps in focusing on achieving quality

• Be key to its success

• Be measurable

• Tell you what you need to know

11

PMs: Best Practice

12

How well did we do…

• Is best practice being implemented

• Is current practice acceptable

Remember…you need to know - how well did we do?

• We use measures to help us determine if what we already know to be the best practice is actually being implemented

• Performance measures help us decide if our current practice is acceptable:o we use measures to screen and/or assess organizational performance

issueso the focus is on the variance from the acceptable standard or threshold

12

Building a Data Dictionary

13

Once you decide on a list of performance measures, you can include them in your Master Data Dictionary.

For the purposes of this training, we have used fictitious data, operational definitions, measure descriptions and some data sources.

We also use example job titles to identify people responsible for data collection, maintenance and reporting. Obviously, job titles vary among organizations and in some cases—especially in smaller organizations—one person may “wear several different hats”.

13

Data Dictionary: Suggested Content

14

Performance

Measure

(PM)

Measure

Description

or Rationale

for Data Use

PM

Numerator:

"Top part of

the fraction"

Parts of the

whole that

meet

identified

criteria being

measured.

PM

Denominator:

"Bottom part

of the

fraction”

The whole

being

measured.

Include

exclusions, if

applicable.

Operational

Definition:

How to

measure a

certain

variable or

how to define

a certain

term to

assure

consistency

in data

collection

and data

analysis.

Additional Notes

Regarding Data

Collection/Analysis

Data

Source

Target

Objective

Person

Responsible:

Data

Collection

and

Maintenance

Person

Responsible:

Reports PM

Findings and

Analysis to

Identified

Audience (s)

1.

2.

3.

Content of a Data Dictionary can be limited or more expansive, depending on an organization’s complexity and/or identified needs. What you see in the table above are ten (10) minimal elements ODP suggests for inclusion in a Data Dictionary.

You may list all of your performance measures and accompanying information on your Master Data Dictionary. It is not necessary to complete a separate dictionary for every performance measure. If you choose to include performance measures that are “counts” only, that is fine. Just know that some of the content elements will not apply (e.g. numerator and denominator).

Make sure staff in your organization, especially those who play a role in the data collection, interpretation/analysis, and reporting processes, are familiar with the dictionary contents. The goal is to make sure your data-related processes are performed in a consistent, accurate manner.

Remember: Data collected and interpreted differently within an organization is data that cannot be used to make meaningful decisions.

In the following slides, we are sharing three different examples of performance measures and accompanying information to populate our Data Dictionary. Taking into consideration the size of the Data Dictionary, we are dividing information for each performance measure into five (5) separate sections—with 2 columns in each section.

14

PM Example #1

15

PM Example # 1

Our first example performance measure counts the number and percent of staff who receive required annual training.

15

PM Example #1: Qualified Staff

16

Performance Measure (PM) Measure Description or Rationale for Data Use

Number and percent of staff

who received required

annual training.

Training of staff enhances

skills, capabilities, and

knowledge of their particular

job. It is continuous in nature,

leads to quality performance,

and promotes the health and

safety of individuals we

support.

PM example #1 - Columns 1 and 2

Performance Measure (PM):

Number and percent of staff who receive required annual training

Measure Description or Rationale for Data Use:

Training of staff enhances skills, capabilities, and knowledge of their particular job. It

is continuous in nature, leads to quality performance, and promotes the health and

safety of individuals we support.

16

PM Example #1: Qualified Staff (continued)

17

PM Numerator:

"Top part of the fraction"

Parts of the whole that meet identified criteria

being measured.

PM Denominator:

"Bottom part of the fraction”

The whole being measured. Include exclusions, if

applicable.

Number of staff who

received required annual

training.

.

Number of staff due for

annual training.


Remember, PM #1 reads: Number and percent of staff who receive training to meet the needs of the Individual they support as identified in the approved Individual Support Plan (ISP) before providing services to the Individual.

The PM Numerator, or “top part of the fraction” - parts of the whole that meet identified criteria being measured:

Number of staff who received required annual training.

The PM Denominator, or “bottom part of the fraction” - is the whole being measured:

Number of staff due for annual training.

For this PM: we will say that 75 staff are due for annual training.

17


18

Operational Definition:

How to measure a certain variable or how to define

a certain term to assure consistency in data

collection and data analysis.

Additional Notes Regarding Data

Collection/Analysis

Staff includes: Direct Service

Workers, Program Specialists

Staff annual training incorporates

the following areas:

• Prevention, detection and reporting

of abuse, neglect and exploitation

• Person-centered practices

• Individual rights

• Safe use of behavior supports

• Implementation of the ISP

Data entered for training year

(previous fiscal year) on the

Staff Training Spreadsheet.

All required trainings must be

completed within the identified 12-

month period of time for each staff

person due for training.


The Operational Definition above reads:

Direct Service Workers, Program Specialists

Staff annual training incorporates the following areas:• Prevention, detection and reporting of abuse, neglect and exploitation• Person-centered practices• Individual rights• Safe use of behavior supports• Implementation of the ISP

Remember, this operational definition is an example, and may not necessarily fit the needs of your organization.

In the Additional Notes:

Data entered for training year (previous fiscal year) on the Staff Training Spreadsheet.

All required trainings must be completed within the identified 12-month period of

18

time for each staff person due for training.

18


19

Data Source Target Objective

Staff Training Spreadsheet

Increase the number of staff

who receive annual training

by 12% (to 75)

by June 30, 2020

FY18-19 Baseline = 67


Data Source:For PM #3, we are identifying the Staff Training Spreadsheet.

In ODP’s QM Certification Module 103, Part 1, you learned that a Target Objective (TO) states the level of performance an organization desires to achieve within a specified period of time and can be established by considering baseline data and benchmarks.

Target Objective (TO):

Increase the number of staff who receive annual training by 12% (to 75) by June 30, 2020

As mentioned previously in slide 17, the number of staff due for annual training is 75. In FY18-19, data results indicated that 67 of 75 staff received the required annual training -resulting in a baseline for FY18-19 of 67 staff (89%). More than 89% of staff should be receiving annual training as required.

You have decided you need to increase the number of staff receiving annual training by 12%, or 8 staff. This would increase the number of staff trained by June 30, 2020 to 75, or 100%.

Calculations are as follows: 67 x 12% = 8; 67 + 8 = 75

You will continue to work on this TO in subsequent fiscal years to ensure that 100% of staff

19

receive required annual training.

19


20

Person Responsible: Data Collection and

Maintenance

Person Responsible: Reports PM Findings and

Analysis to Identified Audience (s)

HR Manager Director of Operations


For our example:

The person responsible for collecting and maintaining data on an ongoing basis is identified as the HR Manager.

This person may have additional staff helping with this process step but he/she is ultimately responsible for making sure data are collected and aggregated on a routine basis for submission to the Director of Operations.

The person responsible for reporting PM findings and analysis is identified as the Director of Operations.

This person is ultimately responsible for outcomes and is in a position (has authority) within the organizational structure to implement needed changes. The Director of Operations will provide routine data reports (e.g. quarterly) to an identified oversight body. Examples of an oversight body include but are not limited to: Board of Directors, Quality Council, etc.

20

PM Example #2

21

PM Example # 2

For this performance measure, we are tracking the number and percent of individuals who receive information about how to identify and report abuse, neglect, and exploitation (A/N/E).

21

PM Example #2: Health & Safety

22


Number and percent of waiver

participants who received

information about how to identify

and report abuse, neglect, and

exploitation (A/N/E).

Providing this information to

individuals gives them tools to

understand and exercise their

rights to be free from A/N/E.

and…

Informs CMS Health and Welfare

Waiver Assurance requirements.



Number and percent of waiver participants who received information about how to identify and report abuse, neglect, and exploitation (A/N/E).


• Providing this information to individuals gives them tools to understand

and exercise their rights to be free from A/N/E

• Informs CMS Health and Welfare Waiver Assurance requirements: The

State demonstrates that it has designed and implemented an adequate

system for assuring waiver participant health and welfare

22

PM Example #2: Health & Safety (continued)

23

PM Numerator:



being measured.

PM Denominator:



applicable.

Number of waiver

participants who received

information about how to

identify and report A/N/E

Number of waiver

participants in the sample


Remember, PM #2 reads: Number and percent of waiver participants who received information about how to identify and report A/N/E.

The PM Numerator, or “top part of the fraction” – parts of the whole that meet identified criteria being measured:

Number of waiver participants who received information about how to identify and report abuse, neglect, and exploitation.

The PM Denominator, or “bottom part of the fraction” --the whole being measured:

Number of waiver participants in the sample

For this PM, we will use a sample size of 315.

23


24






Collection/Analysis

Waiver participants must receive

information about how to identify

and report A/N/E both verbally—in a

private setting—and through

distribution of written materials.

Information may not be shared

with the participant during the ISP

meeting.

Sharing information must be in a private

setting where the waiver participant feels

comfortable disclosing concerns about

allegations of A/N/E—not in meetings or

locations where others who are alleged

targets of A/N/E may be present.

List specific materials intended for

distribution to individuals.

Also note instructions about how completion

of this requirement is to be documented.


If you asked five different people to define in the performance measure what is meant by “received information,” you might get five different answers. To avoid any confusion and to ensure consistency in data collection and analysis, you must reach consensus within your organization about how to define that term.

The Operational Definition above reads:Waiver participants must receive information about how to identify and

report A/N/E both verbally—in a private setting—and through distribution of

written materials. Information may not be shared with the waiver participant

during the ISP meeting.

In the Additional Notes column, we explain in more detail why a private setting is so important:

Sharing information must be in a private setting where the waiver participant

feels comfortable disclosing concerns about allegations of A/N/E—not in

meetings or locations where others who are alleged targets of A/N/E may be

present.

You would also place in this column:

• a list of specific materials intended for distribution to individuals.

Materials to be distributed may change over time, so this information

should be updated accordingly

• instructions about how completion of this requirement is to be

24

documented

24


25


QA&I Review

List specific QA&I

question(s) that will inform

the performance measure

Data collected annually

Increase the number of

individuals who receive

information about how to

identify and report A/N/E

by 30% (to 281)

by June 30, 2020.



Data Source:

For PM #2, we are identifying QA&I Review as our data source. You will also need to identify the specific QA&I question (s) for which data results will be used to inform the performance measure. Frequency of data collection—in this case data will be collected annually.


Increase the number of individuals who receive information about how to identify and report A/N/E by 30% (to 281) by June 30th 2020. FY18-19 Baseline = 216

As mentioned previously in slide 23, the sample size is 315. In FY18-19, data results indicated that 216 of 315 individuals in the sample received information about how to identify and report A/N/E—resulting in a baseline for FY18-19 of 216 individuals (69%). 69% is not acceptable for a number of different reasons.

You have decided to work toward increasing the number of individuals receiving this information by 30%, or by 65 individuals. This would increase the number of individuals who receive information about how to identify and report A/N/E by June 30, 2020 to 281, or 89%.

Calculations are as follows: 216 x 30% = 65; 216 + 65 = 281. You will continue to work on this TO in subsequent fiscal years until you can ensure that 100% of individuals are receiving this critical information.

25


26


Maintenance


Analysis to Identified Audience(s)

QA&I Data Manager QA&I Process Director

And finally…PM example #2 - Columns 9 and 10

For our example:

The person responsible for collecting and maintaining data on an ongoing basis is identified as the QA&I Data Manager.

This person may have additional staff helping with this process step but he/she is ultimately responsible for making sure data is collected and aggregated on a routine basis for submission to the QA&I Process Director.

The person responsible for reporting PM findings and analysis is identified as the QA&I Process Director.

This person is ultimately responsible for outcomes and is in a position (has authority) within the organizational structure to implement needed changes. The QA&I Process Director will provide routine data reports (e.g. quarterly) to an identified oversight body. Examples of an oversight body include but are not limited to: Board of Directors, Quality Council, etc.

26

27

PM Example #3PM Example # 3

The third PM example looks at the number and percent of individuals working in Community Integrated Employment (CIE).

27

PM Example #3: Employment

28


Number and percent of individuals

working in Community Integrated

Employment (CIE)

Executive Order: 2016-03 – Establishing

“Employment First” Policy and Increasing

Competitive Integrated Employment for

Pennsylvanians with a Disability

There is dignity in work, as it provides an

individual not only the income necessary to

meet basic living needs but can help

contribute to his or her self-identity, self-

worth, and self-respect, and offer a sense of

accomplishment; a job can also provide

opportunities for social interaction,

meaningful friendships, and to be and feel

included.



Number and percent of individuals working in Community Integrated Employment (CIE)


Governor Tom Wolf’s Executive Order: 2016-03 – Establishing “Employment

First” Policy and Increasing Competitive Integrated Employment for

Pennsylvanians with a Disability.

There is dignity in work, as it provides an individual not only the income

necessary to meet basic living needs but can help contribute to his or her self-

identity, self-worth, and self-respect, and offer a sense of accomplishment; a

job can also provide opportunities for social interaction, meaningful

friendships, and to be and feel included.

28

PM Example #3: Employment (continued)

29

PM Numerator:



being measured.

PM Denominator:



applicable.

Number of individuals

working in CIE

Number of individuals who

want to work in CIE


Remember, PM #3 reads: Number and percent of individuals working in CIE.

The PM Numerator, or “top part of the fraction” --parts of the whole that meet identified criteria being measured:

Number of individuals working in CIE

The PM Denominator, or “bottom part of the fraction” – the whole being measured:

Number of individuals who want to work in CIE (remember, not everyone chooses to work).

For this PM, there are 89 individuals who want to work in CIE. You know this because there are 89 individuals supported by your organization who have employment as a goal in their ISP.

29


30






Collection/Analysis

CIE is defined as full or part time work at

minimum wage or higher, receiving the

same wages and benefits as co-workers

without disabilities while doing the same

work as and fully integrated in the work

setting with co-workers without disabilities.

A newly hired employee is someone who

either: hasn’t previously been employed in

CIE by the employer, or was formerly hired

by the employer, but has been separated

from such prior employment for at least 60

consecutive days.

Data to be entered into CIE Tracking

Spreadsheet monthly; counting number

of unique individuals newly hired in CIE

as of the end of each month.

It is possible for a person to be

considered newly hired multiple times

during a 12-month period of time.


The Operational Definition above reads:

ODP defines CIE as full or part time work at minimum wage or higher, receiving the same wages and benefits as co-workers without disabilities while doing the same work as and fully integrated in the work setting with co-workers without disabilities.

“Newly hired” is defined as an employee who either: hasn’t previously been employed in CIE by the employer, or was formerly hired by the employer, but has been separated from such prior employment for at least 60 consecutive days.

In the Additional Notes column

Here you would want to make it clear who you are counting and how often your tracking spreadsheet should be populated with new information.

In this case, we are saying data is to be entered into the CIE Tracking Spreadsheet monthly; counting number of unique individuals newly hired in CIE as of the end of each month.

We are also making it clear that It is possible for a person to be considered newly hired multiple times during a 12-month period of time. By collecting this data over time, you will be able to count the number of unique (unduplicated) people with CIE jobs and the number of newly hired/month.

30


31


CIE Tracking Spreadsheet

Data collected monthly

Increase the number of

individuals working in CIE by

20% (to 59)

by June 30, 2020



Data Source:

CIE Tracking Spreadsheet


Increase the number of individuals working in CIE by 20% (to 59) by June 30, 2020. FY18-19 Baseline = 49

As mentioned previously in slide 29, there are 89 individuals who want to work in CIE. In FY18-19, data results indicated that 49 of the 89 individuals who want to work in CIE are working in CIE—resulting in a baseline for FY18-19 of 49 individuals (55%). 55% is not acceptable and you want to work on improving that number.

Since the population of individuals you support live in an area where several new businesses have moved to town, you believe your target objective to increase the number of individuals who work in CIE by 20%--or by 10 individuals--is realistic. This would increase the number of individuals who work in CIE to 59 (66%) by June 30, 2020.

Calculations are as follows: 49 x 20% = 10; 49 + 10 = 59. You will continue to work on this TO in subsequent fiscal years to further increase the number of people working in CIE who choose to work.

31


32


Maintenance


Analysis to Identified Audience(s)

Information Specialist Employment Lead

And finally…PM example #3 - Columns 9 and 10

The person responsible for collecting and maintaining data on an ongoing basis is identified as the Information Specialist.

This person may have additional staff helping with this process step but he/she is ultimately responsible for making sure data are collected and aggregated on a routine basis for submission to the Employment Lead.

The person responsible for reporting PM findings and analysis is identified as the Employment Lead.

This person is ultimately responsible for outcomes and is in a position (has authority) within the organizational structure to implement needed changes. The Employment Lead will provide routine data reports (e.g. quarterly) to an identified oversight body. Examples of an oversight body include but are not limited to: Board of Directors, Quality Council, etc.

32

33

Update your Data Dictionary as needed

• Existing definitions may change

• Add/delete performance measures

• New responsible person(s) are identified

• New data sources

Your Data Dictionary should be updated as needed. For example:

• existing definitions of metrics may change• addition and/or deletion of performance measures• new responsible person(s) have been identified• there may be new data sources identified

33

Conclusion

Thank you for completing Module 2 of the

QM Recertification Course!

34

34

module 2: using a data dictionary · the pm numerator, or “top part of the fraction” - parts of...

Documents