Nova Scotia 811Telecare Service Evaluation Plan

Nursing 5893 Dr. Lori WeeksSteve Iduye, Xiaoping Zhuang, John Amoah-Gyekye.

Agenda

➢Background and Context➢Goal and Objectives➢Target Population➢Evaluation Questions➢Program Theory and Logic Model➢Stakeholders Communication ➢Evaluation Design and Data Collection➢Data Analysis Method➢Resource➢Evaluation Results and Recommendations

Background and Context

➢Inequitable access to health services Disparities in health status Outcomes between rural and remote residents.➢Recent years, tele services have been widely used Telemedicine ： Provide health education➢Healthlink 811： Telehealth programs Nova Scotia Department of Health (DoH) Registered Nurses Over 125 different languages 24 hours a day, 7 days a week

Goal

➢To reduce hospital services utilization through 811 telecare program

➢To evaluate if the program is a catalyst for patients self-directed change in health behaviour

Objectives

➢To improve patients symptoms and Increase patients confidence in self care by

➢To reduce hospital services utilization through 811 telecare program by

95%

10%

Target Population

Residents of Nova Scotia with the following non-emergency symptoms: ➢Abdominal Pain➢Medication Questions➢Chest Pain➢Cough➢Fever in 3 months old or older➢or residents that need to clarify information regarding

medications

Evaluation Questions

➢Does 811 Telecare program reduce hospital utilization services?

➢Is the information provided by Registered Nurses helpful in improving patients symptoms?

811 Telecare Service Program Logic ModelInputs

Activities

Final Outcomes

Immediate Or Intermediate Outcomes

Outputs

✓Budge✓Experienced Registered Nurse✓IT Specialists and Informaticians✓Decision Support Software✓Patients Database

✓Assessment of the urgency of a caller's symptom(s)

✓Provision of health advice and information by experienced registered nurses

✓24 hours per day, 7 days per week operation✓TTY service for the hard of hearing✓Service in over 120 different languages (third-

party language interpretation service)✓Delivery of culturally competent care

✓All residents of Nova Scotia with non emergency symptoms will start to call

✓Improved Callers Symptoms✓Improved Callers Confidence in self care✓Eliminate health knowledge deficit

✓It will improve callers Symptoms and Increased patient's confidence in self-care by 95%

✓It will reduce hospital service utilization by 10%

Stakeholders Communication

• Phase One: The team conveyed our letter of intent to the Telecare service coordinator. We also followed this letter up with several calls in order to establish a relationship with the main program office in Halifax.

• Phase Two: A video conference was arranged by the program office with the team to answer all our questions and to guide us in a direction they think is necessary.

• Phase Three: Our team will deliver the evaluation plan in writing to the program office after we are done.

Evaluation Design

➢In line with the goals and objectives of this evaluation plan, we use:

➢A group pretest and post test quasi-experimental design to conduct the evaluation effect and impact.

Data sources to answer each question posed in the above

• Feedback for caller Satisfaction • Stakeholder Satisfaction from the System• Reactions from the Community• Phone Service Tele triaging impact• 811 Demographic Information• District Health Authorities. • 811 Registered Nurses• Funders

Data Collection Matrix - Matching data collection to key evaluation questions

Key evaluation questions

Interviews Focus Group Questionnaire Case Study

Does 811 Telecare program reduce hospital utilization services?

✔

✔ ✔

✔

Is the information provided by Registered Nurses helpful in improving patients symptoms?

✔

✔

✔

✔

Data Collection Matrix - Data collection (secondary data) options Option What might it include Example

Retrieving existing documents and data

▪Formal policy documents, implementation plans and reports

▪Official statistics Programme monitoring data

▪Programme records

Review of programme planning documents, minutes from meetings, progress reports

Collecting data from individuals or groups

▪Interviews – key informant, individual, group, focus group discussions, projective techniques

▪Questionnaires or surveys – email, web, face to face, mobile data

Key informant interviews with representatives from relevant departments, organizations and/or the wider development community Interviews with programme managers, programme

Limitations of using methods• Interviews

➢ Identifying individuals who are willing to be open to respond to the interview• Focus Group

➢ Scheduling the meeting with the number of people of the focus group. ➢ challenge the difficulty in capturing the information being discussed by the group.

• Questionnaire ➢Some respond statement not clear and needed further clarification. ➢Extreme brief responses may need extra follow up questions making it a bit

difficult to interpret the data

• Case Study ➢An overwhelming amount of data to deal with. ➢Which evaluation questions needs to be refined for a couple of times

Data Collection➢ This evaluation uses a mixed-method (quantitative and qualitative) data collection

approach.

➢ Each evaluation question uses multiple sources of information (triangulation of evidence) to answer the evaluation question. For example, administrative data, information about patient pain and temperature assessment are quantitative data while patient that clarifies medication using 811 teleservice agent generates a qualitative data.

➢ Statistical analysis is necessary to draw conclusions about quantitative data. E.g

Regression analysis is used to predict (including forecasting), inference, and understanding causal relationships.

➢ Qualitative analysis software like NUDIST, ATLAS-ti could also be used in managing and facilitating qualitative data analysis (Issel, 2014)

Data Collection- To Determine Sample Size

➢A power analysis will be conducted in order to determine the minimum sample size needed for this study.

➢By using Gpower 3.1, at expected margin of error of 5%, confidence level of 95%, with initial population size of 137,000 that participated in 811 telecare program last year, and response distribution of 50% with power size of 80%, the minimum sample size for the evaluation is 384.

Data Analysis Method

➢Data is collected and computerized for analysis using a group pretest and post test approach mentioned above.

➢All the data will be coded and statistical calculations will be done using R studio software at the end of data collection.

Data Analysis Method- Change Differences

Pretest (Baseline Measurement): Non emergency symptoms reported by Nova Scotian residents include:

➢Abdominal Pain➢Medication Questions ➢Chest Pain➢Cough➢ Fever in 3 months old baby or older as reported by parents or

caregivers➢No prior hospital or doctor's clinic visit for the same symptoms stated

above

Data Analysis Method- Change Differences

Intervention or Treatment

➢Provision of health information and assistance that is clinically sound, Nova Scotia-appropriate and easily accessible from anywhere in the province.

Data Analysis Method- Change Differences

Post test Measurement

➢Measurement of changes in clinical symptoms stated above (positive or negative changes)

➢Measure the number of referral to hospital or doctor's clinic for the same symptoms stated as baseline

Data Analysis Method- Change Difference Calculation➢ To measure the change difference, the initial pretest baseline is subtracted before the intervention

from the subsequent post test score (Issel, 2014). In other words, the amount of change = sum(each post test- each pretest score) divided by Number of paired scores.

➢ According to Issel, (2014) this formula is good to calculate change difference in only one group,

because each participant difference score is calculated and then averaged across all participants. ➢ For example pain assessment pre and post test could be done by using numeric rating scale(0-10) to

assess level of pain in all the callers with pain symptoms. Pre and post comparison are made.

➢ Cough assessment tool with number awarded to frequency, sound, and duration. Pre and post differences are drawn.

➢ Fever is measured by thermometer with pre and post temperature is compared

➢ Hospital or doctor's clinic visit for the same symptoms stated above is evidenced by the numbers of referral made by the 811 telecare agents.

Resource PlanTask Name Task Description Staffing Expected Costs TimeframeGoal: Evaluating reduction of hospital utilization services through 811 telecare program.Evaluating patients’ health behavior self-directed change

All May 19 -June 30

1. Current State Analysis

Online resource research and contacting clients for more information to Understand the program.

Xiaoqing, 811 staff Internal Human resource,Time,Internet connection,Facilities such as computer and place

May19-May26

2.Developing Evaluation Questions

Create several questions based on program’s objectives for evaluation.

Steve Internal Human resource,Time,Internet connection,Facilities such as computer and place

May19-May26

Resource PlanTask Name Task Description Staffing Expected Costs Timeframe3.Collecting Data Identify what data

should be collected and from who. Data collection methods will be selected.

John, 811 staff, external data source firm staff

Internal Human resource,Time,Internet connection, Facilities such as computer, places, phone call

May19-June2

4.Analyzing Data Analyze collected data with analyzing tools, such as statistical software.

Steve, John, Xiaoqing, professional data analysts

Internal Human resource,Time,Internet connection, Facilities such as computer and place, hire professional data analysts

June3-June12

Resource PlanTask Name Task Description Staffing Expected Costs Timeframe5.Establishing Evaluation results

Based on the data analysis, evaluation results can be created.

Steve, John, Xiaoqing, professional evaluators

Internal Human resource,Time,Internet connection, hire evaluators

June13-June16

6.Developing Recommendations

Based on the evaluation results, recommendations can be made.

Steve, John, Xiaoqing

Internal Human resource,Time, Facilities such as computer and place

June16-June22

7.Presenting Evaluation Results and Recommendations

Presenting the evaluation results and evaluation recommendations to clients.

Steve, John, Xiaoqing

Internal Human resource,Time,Internet connection, Place and facilities for presentation

June23

Evaluation Results and Recommendations

Care Continuum

➢ Finding a means to communicate the patients health needs to the family physician is critical in continuing care. This is made possible through the following measures:

➢Conversation regarding patients health complaints should be forwarded the next day to the patients family physician in text format in the absence of an EHR.

➢811 tele-care database should continuously interfaces with electronic health record for easy information accessibility by other care providers.

Evaluation Results and Recommendations

Upgrade Care

➢Develop application which can be used for both voice contact and video contact such as Facetime, Skype.

➢Patients with their own health ID which connected to their health records to sign in this 811 application.

➢Patient with such app can show their symptoms and give more information to the registered nurse and that can improve healthcare delivery.

References

➢ Issel, L. (2014). Health program planning and evaluation : A practical, systematic approach for community health (3rd ed.). Burlington, MA: Jones & Bartlett Learning

➢ Hill, Kimberly Anne. (2014, March). Determining the Impact of Telehealth on Rural and Remote Health Care Service Delivery in Canada. 2014 University of Regina. Retrieved from: http://ourspace.uregina.ca:8080/handle/10294/54897

➢ S. Mahadevan, M.B.A., M.Phil., K. Muralidhar, M.D., M.B.A., and D. Shetty, M.S., F.R.C.S. (2012, November). Tele-Education Service Using Telemedicine Network in Healthcare Industry. TELEMEDICINE and e-HEALTH . MARY ANN LIEBERT, INC. VOL. 18 NO. 9.

➢ McKesson Canada. (2010, July/August). HEALTHLINK 811. Nova Scotia partners to provide better access to care. Retrieved from: http://www.summitconnects.com

➢ For trusted, everyday health information and advice, call 811. Retrieved on 2016, May 29 from: https://811.novascotia.ca/

Questions?