healthdirect australia research and evaluation …...research and evaluation strategy aligns to...

Healthdirect Australia

Research and Evaluation Strategy

2014-18

October 2014

Healthdirect Australia Research and Evaluation Strategy 2014-2018 2

Contents Contents ......................................................................................................................... 2

Introduction .................................................................................................................... 4

Building the research and evaluation agenda ............................................................... 4

Development of the strategy ......................................................................................... 5

Part 1: Aligning to Healthdirect Australia Strategic directions ................................... 5

Research and evaluation as an integral part of Healthdirect’s work program ................ 8

Aims of the research and evaluation strategy ............................................................... 9

Part 2: Research and evaluation governance ............................................................ 10

Purpose ...................................................................................................................... 10

Aim ............................................................................................................................. 10

What we will do .......................................................................................................... 10

Oversight .................................................................................................................... 10

Funding ...................................................................................................................... 11

Research governance ................................................................................................ 11

Part 3: Setting research and evaluation priorities ..................................................... 13

Purpose ...................................................................................................................... 13

Aim ............................................................................................................................. 13

What we will do .......................................................................................................... 13

Healthdirect research and evaluation priorities for 2014-15 ........................................ 15

Research, evaluation and monitoring priorities ....................................................... 15

Part 4: Develop and implement an evaluation framework ........................................ 16

Purpose ...................................................................................................................... 16

Aim ............................................................................................................................. 16

What we will do .......................................................................................................... 17

Routine collection of data ........................................................................................... 20

Using the results of evaluation studies ....................................................................... 20

Part 5: A framework for responsive research ............................................................ 22

Why research is important for Healthdirect ................................................................. 22

Aim ............................................................................................................................. 22

What we will do .......................................................................................................... 22

Partnering in research ................................................................................................ 22

Commissioning researchers ....................................................................................... 23

Applying Criteria for selecting research partners ........................................................ 23


Managing requests for data and research collaborations ........................................... 23

Commissioning research or reviews of research ........................................................ 23

Part 6: Communications .............................................................................................. 25

Purpose ...................................................................................................................... 25

Aim ............................................................................................................................. 25

What we will do .......................................................................................................... 25

Appendix 1 ................................................................................................................... 26

Levels of Monitoring and Evaluation ........................................................................... 26

Appendix 2 ................................................................................................................... 27

Evaluation and monitoring framework - Healthdirect services and programs .............. 27

Appendix 3 ................................................................................................................... 29

Key drivers for evaluation ........................................................................................... 29

Introduction

The National Health Call Centre Network (NHCCN) trading as Healthdirect Australia

(Healthdirect) is a company established by the Council of Australian Governments

(COAG) which strategically procures publically funded telehealth services.

The following services are currently provided inter alia through Healthdirect:

Healthdirect – nurse triage line

After hours GP Helpline

Pregnancy, Birth and Baby, Counselling service and website

mindhealthconnect

National Health Services Directory

My Aged Care Gateway (website and contact centre)

Healthdirect (website)

Get healthy Service

As and when new services come on board they will be incorporated into the research

and evaluation strategy.

Building the research and evaluation agenda

As Healthdirect continues to embed its services into the Australian health system there is

now a recognised need for a long-term strategic plan that addresses Healthdirect’s

research and evaluation agenda.

This involves ensuring that Healthdirect participates in relevant and timely research and

evaluations to continue to create services that are innovative and evidence-based,

technologically sound, address the needs of both consumers and the Australian

healthcare system, are appropriately targeted, and provide value for the resources

invested.

As Healthdirect’s core business is providing health services rather than being a research

entity, the priorities for research will be driven by business and service imperatives. This

Research and Evaluation strategy aligns to Healthdirect Australia’s organisational

strategy (2014-2018). As it seeks to strengthen its reputation and expertise in order to

be:

An evidence-based service provider of quality programs that improve health

outcomes

A contributor to the body of knowledge about the effectiveness of telehealth and

online health services and how to deliver these to reach all the Australian

population

An innovative leader in telehealth and online health services

An authoritative voice on future directions and investment priorities in telehealth

and online health services.



Development of the strategy

The research and evaluation strategy (the Strategy) was developed on the basis of

consultations and discussions with Healthdirect executives, the Clinical Governance

Advisory Committee, the Joint Customer Advisory Committee and the Digital Services

Advisory Committee. In addition a workshop was held with key staff, shareholders and

stakeholders to consolidate summarise and synthesise current activities and directions,

set new priorities for the future, and agree on the infrastructure and processes that are

needed to set these in place.

Part 1: Aligning to Healthdirect Australia Strategic directions

Healthdirect has recently moved into a longer term funding cycle and is accordingly

developing new strategies to advance work into the future. The Research and Evaluation

Strategy 2014-2018 aligns with the Healthdirect strategic plan (Figure 1) and follows the

current NHCCN Research and Evaluation Strategy 2012-2014.

The process of ensuring this strategy has aligned to the Organisational strategic plan has

resulted in the development of six strategic initiatives and a range of actions described in

Figure 1. These six strategic initiatives, along with the actions, have been developed as

resources to provide guidance to Healthdirect staff when planning and managing

research and evaluation projects. Further details about these actions are described

throughout the document.


To be a key part of quality Australian

health system by helping consumers

manage their own health through

leveraging technology to enable timely

access to health and related services

To provide a set of principles and activities

that can be used to support the delivery of

high quality research and evaluation which

is relevant and responsive to the

Healthdirect Australia strategic directions.

Align to Healthdirect

Strategy

Research and evaluation

governance

Setting research and

evaluation priorities

Develop and implement

robust evaluation framework

Develop and implement

framework for responsive research

Commun

ication

Identify, develop solutions, aggregate,

procure and evaluate publically funded

health and related services delivered through

innovative technologies

Healthdirect

Australia Vision

Strategic

initiatives

Ensure evidence-based

policy and services

Apply best practice

principles

Enable continuing

improvements in service

delivery and targeting

Deliver products and services

that are competitive and world leading

Ensure clinical appropriateness

of services

Measure cost effectiveness /

value for investment

Justify future funding for

research and evaluation

Purpose

Research and

evaluation

strategy aim

Figure 1. The Framework

1 2 3 4 5 6


Develop and enhance multi/cross-platform initiatives for relevant services to new channels and technologies

services

Deliver evidence-based, best practice clinical go services

Enhance current portfolio of service

Develop new services Facilitate health relationships Enhance organisationalcapability

• Develop and enhance multi/cross-platform initiatives for relevant services

• Expand relevant services to new channels and technologies

• Develop and enhance call services • Deliver and enhance online

services• Deliver evidence-based, best

practice clinical governance of all services

• Provide strategic leadership for new initiatives

• Investigate identified health related services opportunities

• Enhanced medication information services

• Improve health literacy• Integrate video consulting

capability into new services

Promote and initiate consumer initiatives

Build evaluation capacity across the organisation

Part 2: Research and evaluation governance: Implement a research governance framework

Part 2: Research and evaluation governance: Implement a research governance framework

Part 5: Develop and implement a framework for responsive research: Partner with the most suitable research organisations, with high quality researchers to undertake exploratory research

Part 2: Research and evaluation governance: Establish and maintain an organisation-wide steering group.Seek endorsement for research and evaluation concepts and projects through CGAG and PROWAC

Part 3: Setting research and evaluation priorities: Establish andfacilitate a process for identifying and updating Healthdirect research priorities. Through the use of a triage system.

Part 3: Setting research and evaluation priorities: Ensure funded research funded aligns to Healthdirectpriorities . Determined through the Advisory Group and approved by CGAG and PROWAC

Part 6: Communications: Promote research and evaluation to the wider health and research community through a range of mediums Every research project will have a company communication plan

Part 4: Develop and implement a functional evaluation framework: Encourage a culture that supports the use of research and evaluation findingsDisseminates findings through Company committees and PROWAC

Part 4: Develop and implement a functional evaluation framework: Ensure evidence, evaluation and monitoring data is used to enhance program effectiveness. The Advisory group will be used to facilitate this process.

Part 4: Develop and implement a functional evaluation framework: Strengthen organisational evaluation capacity and processes

Ensure research and evaluation promotes equity and health literacy.

Part 5: Develop and implement a framework for responsive research: Commission high-quality evidence checks to ensure technology enhancements are based upon the evidence available.

Part 5: Develop and implement a framework for responsive research: Partner with the most suitable research organisations, with the most suitable expertise to undertake exploratory research

Organisational Strategic

pillars

Initiatives that

require a research

and evaluation

function

Research and

evaluation actions –what we will do

Figure 2. Strategic alignment to organisational goals


Research and evaluation as an integral part of Healthdirect’s work program

This agenda is critical to the ongoing integration of

Healthdirect into Australian health system. Aligned to

the Healthdirect Clinical Governance framework this

Strategy drives:

1. Research that informs management decision

processes about new services, service gaps or

changed circumstances that can ultimately

influence Healthdirect services.

2. Evaluation to manage the process of

continuous quality improvement (CQI). To

ultimately understand impact and improve the

efficiency and effectiveness of existing services

The following are key reasons for Healthdirect Australia

to prioritise research and evaluation:

Ensure evidence-based policy and services

Apply best practice principles

Enable continuing improvements in service delivery and targeting

Deliver products and services that are competitive and world leading

Ensure clinical appropriateness of services

Measure cost effectiveness / value for investment1

Justify future funding for research and evaluation

Be seen as a leader in tele-health.

Healthdirect is committed to:

Embedding research and evaluation as an

integral part of the organisation’s approach to

program and service development and delivery;

and

Ensuring that the results are considered,

disseminated and utilised.

1 Note that there are several opportunities for measuring value, with cost–effectiveness being just one. Assessments of cost utility, benefit,

efficiency or even appropriate cost-shifting may be relevant in different instances. 2 http://www.dpc.nsw.gov.au/__data/assets/pdf_file/0009/155844/NSW_Government_Evaluation_Framework_Aug_2013.pdf 3 http://www.innovation.gov.au/Research/ResearchBlockGrants/Pages/HigherEducationResearchDataCollection.aspx

Monitoring2 is a management

process to periodically report against

planned targets (or KPIs). For the

most part, monitoring is not

concerned with questions about the

purpose, merit, or relevance of

services. Monitoring is frequently

based on outputs as opposed to

outcomes.

Evaluation2 is the systematic

collection of information about the

activities, characteristics, value and

outcomes of services to make

judgements about the services’’ merit

or worth, usually in relation to their

effectiveness, efficiency and

appropriateness. The information

gained is used to improve program

effectiveness and /or inform

decisions about future programming.

Research3 is the creation of new

knowledge and/or the use of existing

knowledge in a new and creative way

so as to generate new concepts,

methods and understandings.

The primary difference between

research and evaluation is that

research is conducted with the intent

to generalise the findings from a

sample to a larger population. In

contrast, evaluation particularises,

focusing on an internal situation with

no intent to generalise the results to

other settings and situations.

Research informs management

decision processes about new or

enhanced services to meet service

gaps or changed circumstances.

Evaluation guides management in the

process of CQI to improve existing

services in terms of efficiency and

effectiveness.

http://www.innovation.gov.au/Research/ResearchBlockGrants/Pages/HigherEducationResearchDataCollection.aspx


Aims of the research and evaluation strategy

The overarching aim of the strategy is to provide a set of principles and activities that can be

used to support the delivery of high quality research and evaluation that is relevant and

responsive to the Healthdirect Australia strategic directions and support a consistent

approach to enhancing research and evaluation. The following sub aims have been

developed to support this.

To develop and implement research processes that can:

- Identify gaps in knowledge and enable problem identification and the

development of evidence based solutions to those problems

- Recognise and respond to health and healthcare priorities, rapidly changing

technologies, and patterns of consumer behaviour with respect to these

technologies

- Support priority setting and future directions for research

- Enable the translation of research findings into actionable performance

improvement measures.

To determine criteria and guidelines that can:

- Enable the identification and evaluation of possible research partnerships

- Position Healthdirect as a recognised and valuable research partner.

To develop and implement evaluation measures that can:

- Assess impacts of Healthdirect services on individual health, population health

and the healthcare system

- Recognise local area and specific sub-population needs

- Include measures of cost-efficiency, cost-effectiveness and cost-shifting

- Enable Healthdirect to state with confidence the effectiveness, efficiency and

safety of its services

- Be utilised to improve services.

To ensure that Healthdirect services meet the needs of all Australians and are

culturally sensitive to those needs.

To measure the demand for current Healthdirect services and assess how this could

be improved, particularly with respect to equity and access of healthcare.

To identify and describe the infrastructure, governance and resources that will assist

Healthdirect’s work in research and evaluation and ensure this work is of high quality

and recognised.

To investigate similar international services to identify strategies that can be

translated into the Australian model.


Part 2: Research and evaluation governance

Purpose

The following section sets out the purpose and principles for the governance of the research

and evaluation strategy.

Good governance firstly ensures accountability and transparency of decision-making. This

will include introducing a standardised and streamlined approach to the development of

research and evaluation briefs to ensure they align to organisational needs. Secondly, it

ensures the implementation of good practice principles. This includes ensuring that NHMRC

ethical principles of research and privacy principles are adhered to so participants/users of

Healthdirect services are not put at risk.

Overseeing the delivery of this strategy will involve building evaluation capacity across the

organisation. This will be achieved by actively encouraging engagement, knowledge transfer

and collaboration across divisions and external agencies. Additionally, this supports the

production of new knowledge from research/evaluation which is based upon clearly defined

questions relevant to Healthdirect services, underpinned by rigorous and valid research

methods and of publishable quality.

Aim

To ensure all research and evaluation studies adhere to best practice guidelines in accordance with national research bodies and Healthdirect quality assurance and governance processes.

What we will do

Oversight

The strategic oversight of the research and evaluation strategy will be undertaken by the:

Clinical Governance Advisory Group (CGAG): It will be the role of this

Committee to endorse research and evaluation projects, and provide clinically

relevant advice as and when required.

Project Review and WHS Advisory Committee (PROWAC) In accordance with

PROWAC scope of responsibilities any research and evaluation project which

meets the PROWAC criteria will be reviewed by this committee.

Portfolio Management Office: If projects are of a substantial size and link to

business case development the PMO will be informed and involved in accordance

with their Terms of Reference

To manage the implementation of the strategy, we will establish an internal steering group

led by the Clinical Governance Division. The role of this group is to:

Triage and approve proposals against the criteria for identifying suitable Healthdirect

research

Prepare research and evaluation proposals for review and acceptance by CGAG

Ensure research governance principles are adhered to

Assess the risk of the proposals

Monitor the execution of research or evaluation studies

Provide guidance on the use of research and evaluation results

Come together to resolve issues and share best practice


This group will meet bi-monthly to oversee the implementation of this strategy and will

provide monitoring for existing research.

Funding

Some evaluations or small research/ need assessment projects will be undertaken as

Business as Usual (BAU). Any other research and large scale evaluations will require

business case development and funding from Commonwealth, jurisdictional Government or

another source. It will be the responsibility of the steering group to approve these business

cases before they are reviewed by other appropriate channels.

Research governance

A research governance framework will be developed and implemented across the

organisation to develop a standard process for procuring and monitoring research projects.

This will include a process which defines and communicates good practice principles and

requirements. In summary, the framework will provide a structure for:

Sound management of routinely-collected data that are secure and quality-assured

Guidelines and IT infrastructure for making routinely-collected data available to

researchers upon receiving approvals

Process for gaining ethics approvals

Resources, training and capacity for managing grants

Resources training and capacity for managing research relationships and

partnerships

Access to ethics committees and resources for managing ethics approvals

Policies to guide decisions on publication and intellectual property and moral rights

Procedures for managing instances of research misconduct.

The implementation of these actions will be discussed at the Research and Evaluation

Steering Group.

As a Joint responsibility between the divisions of Clinical Governance and Marketing and

Strategy a database will be developed and record all research and evaluation studies that

have been led by Healthdirect or in which Healthdirect has participated.


Table 3 identifies the principles that will be included in the Research Governance Framework

to inform Healthdirect’s decisions regarding research and evaluation partnerships and similar

arrangements2.

2 Adapted from the principles for research governance developed by the Sax Institute 2014

Table 3. Key Principles

For any research and evaluation partnership or similar arrangement, the following principles may apply:

a) A financial arrangement should not influence, impose or imply conditions that would limit, or appear to limit, Healthdirect’s ability to carry out its functions fully and impartially, with integrity and in accordance with high professional standards.

Healthdirect will not engage in relationships where third parties seek to inappropriately influence the content of research outputs or the means by which evidence from research is provided to policy-makers, in order to further their own private interests.

We will maintain the highest academic standards and abide by all applicable research codes of conduct and ethics.

We will not enter into any relationships which hinder our ability to act with integrity in accordance with such standards and codes of conduct.

There should be no real or apparent conflict of interest between the mission and objectives of Healthdirect and those of a funder or partner.

Healthdirect will not enter into arrangements with organisations whose core business is in any way contrary to the mission, or that are deemed not to operate in the best interests of improving health.

b) Healthdirect will seek to avoid any potential risks to protect public confidence in the company and its reputation.

Healthdirect will not engage in any form of private sector arrangement which would seek to utilise our resources for purposes that are not in the public interest.

c) All private sector arrangements will be documented in writing and/or set out in a formal contract.

The terms and conditions of all arrangements will be documented. This may include emails, letters of agreement, and/or formal contracts, depending on the nature and value of the arrangement. Verbal agreements must be noted in writing for the record.

The elements which must be documented are the agreed fees, conditions and benefits, Company and sponsor brand management protocols and the life time of the agreement.

All funding arrangements are to be endorsed by the relevant program head and approved by the Chief Executive Officer. Board advice will be sought as required.

As Healthdirect seeks to improve health, and health services and programs, organisations that are involved in activities and/or products that present a direct hazard to community health and wellbeing are regarded as being inappropriate for any form of sponsorship. Some examples include companies involved in the production and sale of tobacco and alcohol products.


Part 3: Setting research and evaluation priorities

Purpose

To ensure that research and evaluation are driven by Healthdirect business and service

imperatives, a “themes” approach to setting research and evaluation priorities will be taken.

Researching health priorities may not always be relevant to Healthdirect consumers,

therefore key topics and priorities will be based upon the Company, shareholder and

consumer needs. These “themes” will be reviewed annually in conjunction with the review of

divisional business plans. Development of the themes will be based on:

National health needs

Emerging issues in telehealth and e-health

Driving best practice

Evaluation data and analyses

Consultations with shareholders and stakeholders

Information from pilot and demonstration programs

Healthdirect’s Strategic Plan.

Healthdirect’s current themes for its research and evaluation priorities are:

Ensuring equity of access to appropriate health advice and services provided

by Healthdirect –including reaching populations with difficulties accessing services

because of language, physical access and cultural needs.

Empowering users - enabling users to make the most effective use of the services

and information provided by Healthdirect in accordance with their needs and

expectations. This would include improving health literacy.

Coordination and integration – helping people with chronic disease and disabilities

(and their healthcare providers and carers) to better manage their conditions.

Improving technology literacy - utilising new technologies effectively where they

are available and supporting people to use accessible technologies most effectively.

Economic analyses and impacts – assessing cost benefits, cost-effectiveness, cost

shifting and value in terms of outcomes both for consumers and for Healthdirect.

Aim

To optimise research conducted through Healthdirect by ensuring it aligns to organisational

goals and commitments.

What we will do

Establish and facilitate a process for identifying and updating Healthdirect priorities and

assessing research and evaluation studies against the identified key themes.

This process will involve proposed research studies being triaged against criteria (Table 2)

for identifying suitable Healthdirect research and ensuring adequate probity for partnerships

(Table 3). These will be used to ensure:


All research and evaluation studies are relevant and structured around Company

priorities as described in the Healthdirect Australia Strategic plan.

Collaboration with external shareholders and agencies to translate and integrate

research findings into new services or enhance existing services.

Research and evaluation outcomes are communicated in the most suitable format.

Table 4. Criteria for identifying and approving Healthdirect research proposals:

The proposed research or evaluation aligns to the key themes described above

The proposed research is relevant and aligns to Healthdirect and shareholders’

strategic objectives

The research addresses a gap in knowledge, adds value and is timely

The research findings can contribute towards growing the evidence base

The proposal includes sound methods that are relevant and responsive

Offers a return on investment through the use of results evidence to grow new

services or enhance existing Healthdirect services

The proposed research addresses health inequities and improving health

literacy

Provides opportunities for partnerships between (researchers, Healthdirect and

policy-makers)


Healthdirect research and evaluation priorities for 2014-15

These priorities were developed on the basis of current research being undertaken,

consultations with senior management, a review of literature on current issues and through a

questionnaire to workshop participants. This list is not exhaustive, additional research and

evaluations will be undertaken as and when new initiatives or opportunities arise.

Research, evaluation and monitoring priorities

Research Conduct concept testing for new telehealth care delivery models for mental

health in nurse triage

Increase understanding, measure and improve health literacy through

Healthdirect services

Test the effectiveness and cost effectiveness of remote monitoring for chronic

conditions

Concept testing new digital delivery models for disability services

Evaluation Develop a clinical scenario testing tool for the After Hours GP Helpline and

nurse triage service

Conduct periodic clinical audits of Healthdirect nurse triage service and AGPH

Evaluate the acceptability, appropriateness and use of the Healthdirect

website

Conduct a longitudinal study on the effectiveness of video consultation

Track cohorts of users through the Primary care linkage program – assess

usability and outcomes

Assess the impact of the Symptom Checker on consumer satisfaction and

outcomes

Monitoring Monitor the usability of Symptom Checker and outcomes

Use Healthdirect data as an early warning surveillance system for infectious

diseases

Develop a framework to measure access and effectiveness of digital services

part of an ongoing monitoring

In addition to research, evaluation and ongoing monitoring a range of feasibility and needs

assessment studies will be conducted to collect and analyse information to inform future

services or business proposals. These include:

E prescribing for general practitioners on the Afterhours GP Helpline

Evaluation of current services for providing medication information – telephone and

digital services,

Primary care alternatives to emergency departments.


Part 4: Develop and implement an evaluation framework

Purpose

When an evaluation is planned and conducted well, it can provide timely evidence on service

effectiveness and efficacy. This evidence can be used to improve services and provide

accountability and transparency to shareholders and consumers.

Evaluation can play a key role in testing the impact of new services and the ongoing

effectiveness of mainstream Healthdirect services. To achieve this, we aim to ensure that an

evaluation plan is developed concurrently with service business plans and funding to conduct

evaluations is built into the business planning process. This will facilitate an evaluation

process that can be used throughout the lifecycle of a program.

To help gather evidence that meets its intended goal, evaluations can be classified as

formative, process or summative on the basis of:

1. The operative stage of the service (this can include development, recently

implemented, long-term)

2. The purpose of the evaluation (e.g. can the program be improved, is it well-targeted;

is it cost effective)

3. How the information obtained will be used (e.g. to support the implementation of a

new program or the expansion of an existing one; to justify continued funding for a

program).

Monitoring and routine data collection can also be included as a preliminary step.

Aim

To introduce a clear program of change that can be used to improve evaluation outcomes.

To ensure each evaluation adds value, the following functional framework will be introduced

across the organisation as a structured approach to enhance:

a) The planning phase of an evaluation to ensure that an evaluation achieves its

intended goal, and

b) Building the capacity of Healthdirect staff when participating in the design and

implementation of service evaluations3.

This is important because it:

Ensures that evaluations are appropriately funded

Evaluation objectives and outcomes are properly identified

Guides consistent, transparent and timely evaluations

Ensures that evaluations of individual programs

- work toward a common purpose

- are complementary

- address questions important to the organisation as a whole

Identifies the level of inquiry

Identifies high-level outcomes.

3 Healthdirect Australia, Strategic Plan 2014-2018. Pillar 4. Promoting Organisational capability, Business plan initiative No 16:

Enhancement of services though continuous development of research and evaluation capacity and strategy.


What we will do

To strengthen the evaluation capacity conducted through Healthdirect, in this section we

describe seven steps involved in developing an evaluation brief. These steps, which have

been adapted from the NSW Government Evaluation Framework aim to support an

organisational approach to achieving this:

STEP 1: Build an evaluation framework into service design

Planning an evaluation before a service is implemented provides better outcomes. This will

be achieved by:

Building evaluation objectives into the business planning phase

Developing a program logic model that explains how the intended outcomes relate to

the activity, together with measurable performance indicators and outcomes and

interim measures

Articulating evaluation questions relevant to the evaluation purpose

Identifying data needs and key stakeholders, if necessary comparison data

Engaging to most appropriate stakeholders.

Figure 4 provides a flowchart for managing the execution of an evaluation study

STEP 2: Apply a Framework to streamline the planning of an evaluation

The evaluation framework described in Appendix 2 will be used, as and when appropriate, to

streamline the evaluation planning process. The purpose of the framework is to provide a

menu for assessing, monitoring priorities, determining what is feasible and ensures

consistency across evaluations. Key drivers for evaluation are shown in Appendix 3.

Using a key drivers approach, the framework identifies nine areas of interest

specifically relating to the implementation and anticipated outcomes of Healthdirect

services. Depending on the program or service to be evaluated, individual drivers will

have greater or lesser prominence.

Using level of monitoring and evaluation approach, the framework identifies different

types of evaluation that may be applied to each of the nine key areas. As a general

rule:

o All programs and services should be subject to input/output monitoring

o Most to all services should be subject to process evaluation

o Emerging services should be subject to formative evaluation

o High impact, new, and important services and programs should be subject to

summative evaluation.

The framework can be used as a guide in setting key priorities and key questions for each

evaluation and each field in the framework should be considered for its relevance and

appropriateness for the service or program being evaluated.


STEP 3: Ensure evaluation methods are rigorous with appropriate scale and

design

This will be achieved by:

Employing data collection methods that are most appropriate to the research

objectives and program operations.

Using best practice methods and relevant data sets which are reliable and can be

accessed for repeated evaluation.

Since resources and time are required to conduct evaluations, they should be designed and

scaled to each program in accordance with the program’s size, risk, and significance.

STEP 4: Use the right mix of expertise and independence

Healthdirect may commission a suitable agency to conduct an evaluation on behalf of the

company, judgements about the most suitable agency will be based upon the purpose of the

evaluation, questions that need answering and resourcing. This will be undertaken through

the Healthdirect Procurement process.

To add to the body of knowledge relating to each service area program managers, other staff

and stakeholders involved in service delivery will be encouraged to participate in the planning

and review of recommendations of the evaluation.

STEP 5: Ensure the planning of evaluation includes an appropriate timeframe

Evaluations should conclude before decisions are made, with consideration given to the

realistic amount of time needed to conduct them to answer the questions being asked.

Summative evaluations should not be undertaken too early, in recognition of the time it can

take to accrue sufficient evidence and produce measurable outcomes.

STEP 6: Recognise the context in which the activity is conducted

Evaluations should understand how the political, social, economic, technological and cultural

contexts have affected the program being evaluated and the people who are using it.

STEP 7: Apply good governance

Comprehensive information on all aspects of the evaluation should be systematically

recorded, including choice of methods, analyses and conclusions. The extent to which

evaluation findings, methods and data are shared within and external to Healthdirect should

be determined during planning and be transparent and open to scrutiny (as described in Part

2).


Figure 4. A flow chart for managing evaluation studies

Describe of what needs to be evaluated

• Program description

• Program logic model

• Consultation with evaluation framework

Engage and participate

• Need involvement of program managers, experts, evaluators (internal and external), stakeholders and those who need / will use the findings

Develop evaluation brief

• Purpose, scope, key evaluation questions, timelines, available resources

• Review questions against evaluation framework

Develop of evaluation design

• How data will be collected, analysed and reported to answer key evaluation questions

• Ensure statistical significance / meaningful results

• Match design and results to significance and risk of program to be evaluated

Develop of work plan

• Timeline for deliverables, responsibilities

• Agree on communication, publication issues and risk management of findings

Manage implementation of

work plan, and report/s

• Timeline for deliverables, responsibilities

Disseminate report/s and use

of findings

• Recommend for policy and /or action


Routine collection of data

Part of the planning of evaluation is to ensure adequate comparisons can be made. For this

reason, processes for ensuring the necessary data are collected, in a way that is efficient

and not burdensome. In driving the decisions around data collection, the following will be

considered:

What is currently measured

What needs to be measured

What evaluation studies have been completed / are currently underway

What can be evaluated through quantitative data and what needs to be assessed

through qualitative data

What aspects should be regularly evaluated (to give longitudinal data)

How to cross link information from the various evaluation studies

Usefulness of the data.

It is necessary to ensure that sufficient data are collected to deliver statistical significance to

the final results and this factor might affect how some low frequency or highly targeted

services are best measured.

Decisions about routine data collection would be the responsibility of the internal research

and evaluation steering group.

Using the results of evaluation studies

To ensure the results of an evaluation are fed back into the organisation in an efficient way,

staff and stakeholders will be regularly provided with relevant and useful information that can

be utilised in service changes, foresight discussions and communications and public

awareness initiatives.

In the first instance this will be through the use of Company intranet “Pulse” and through

existing Healthdirect Committees.

This will support a continuous evaluation process (sometimes referred to as a ‘benefit cycle’

or a ‘virtuous cycle’) whereby evaluation data are regularly assessed and findings are fed

back into programs to improve functioning, targeting, efficiency and outcomes.

Figure 5 below represents the dynamic process of evaluation built around the support and

effective response of the organisation in undertaking an evaluation as part of its core

business activities and responding effectively to evaluation findings. The process is

described in circular fashion as it is a constant process of review – analysis - implementation.


Organisational

support and response

Figure 5 The continuous evaluation cycle

Engage stakeholders

Ensure evaluation design fits the program

Gather and analyse evidence

Justify conclusions in

light of program goals

Ensure use and share lessons

learned

Program adjustments and / or research to

address information

needs


Part 5: A framework for responsive research

Why research is important for Healthdirect

Effective health research can contribute to identifying business development opportunities,

service enhancements and professional development opportunities for call centre and digital

staff. This can result from understanding the trends in customer needs and service

opportunities that will improve Healthdirect’s performance over time. Investment (and

disinvestment) decisions should be guided by the outcomes of the research service.

A key aim of this Strategy is to ensure a sound research structure that will have the capacity

to:

Identify gaps in knowledge

Recognise and respond to health and

healthcare priorities, rapidly changing

technologies, and patterns of consumer

behaviour with respect to these technologies

Facilitate priority setting and future directions

for research

Enable the selection and evaluation of

possible research partnerships

Ensure the success of research

collaborations and partnerships.

Aim

Optimise the use of research to inform existing and new services

What we will do

Ensure that all funded research provides outcomes that can be used to enhance existing

services or provide the evidence to develop new services. This will be achieved through:

Partnering in research

Partnering in research provides Healthdirect with access to existing capacity, knowledge and

resources across different organisations and settings and brings these together for high

quality, resource-efficient research. This approach provides opportunities to work

collaboratively with academic partners to answer questions important to the delivery of

Healthdirect’s services. Working collaboratively also provides research partners with the

ability to use large data sets to answer important questions on the delivery of health services

to Australians. This may include research on emerging technologies that may be of

importance to the delivery of Healthdirect’s services, or on topic areas that have been

determined to be priorities for Healthdirect.

Partnerships may be developed for individual research projects, or may be established

through a memorandum of understanding or similar for collaboration over a longer period of

time and may include several or more research projects. Partnerships may be initiated by

Healthdirect, or by another organisation including a university, research institute, or other

agency.

What is responsive research?

Research that: 1. Responds to community

needs, shareholders’ priorities and knowledge gaps, and;

2. Can help drive and improve Healthdirect’s services.


Building research partnerships may involve a proactive search for research partner/s or a

reactive response to a request for Healthdirect to be a research partner. In either case this

will involve:

Determining and refining research question/s

Determining study design

Gaining ethics approvals

Applying for funds, holding funds

Agreeing on intellectual property, moral rights and publication of results.

Commissioning researchers

Through the development of the research triage system (Part 3) and the establishment of a

research database, we will endeavour to commission research organisations that are most

appropriate for research initiatives. Commissioned research will generally be carried out for:

Reviews and analysis of research areas

When answers are needed to a specific question

When an independent process is desirable or required.

Where Healthdirect is interested in knowing what the existing evidence for a particular

question is, commissioning a review of existing literature will be undertaken.

Applying Criteria for selecting research partners

All applicants to conduct commissioned research and all potential research partners will be

assessed against Healthdirect’s criteria for selecting research partners (see the principles

established in Part 2).

Managing requests for data and research collaborations

As and where applicable support external agencies to access and use Healthdirect data and

provide responses to data requests in a timely and transparent fashion. This includes:

A clear process for submitting requests

An agreed timeframe and procedure for dealing with such submissions

Designated responsibilities for managing this procedure

Consideration of all requests on the basis of Healthdirect’s research priorities and

guidelines for research partnerships.

See Figure 6 for an outline of how such requests will be managed.

Commissioning research or reviews of research

There will be occasions when Healthdirect’s research needs can only be met by

commissioning research or reviews of research from external experts.

Procedures for commissioning research will require development of appropriate mechanisms

to manage research commissioning processes. This will include agreement on roles and

responsibilities of the business units and the research and evaluation advisory board,

including decisions on:

- Funding and resources approvals

- Appropriate procurement methods

- Selection of research teams


Figure 6: A flow chart for managing research requests

1. Submission portal

•Clear point or points of entry for requests for data and research partnerships

•Contact details for Research Manager to assist discussion

2. Initial consideration

• Regularly scheduled meeting of research and evaluation advisory board

• Consider request in light of criteria for research partners, established priorities, existing work and resources

• Decision to proceed or deny

• Response back to party making the request within a stated timeframe

3. Consultation

May involve external experts

• Decision on research design, timeframes, deliverables

• Agreement on responsibilities, additional partners if needed

• Agreement on funding and resources

• Agreements on intellectual property, moral rights , publication of results as appropriate

• Legal consultation

4. Application for funding (if applicable)

•Joint agreement on development of funding application

•Response to reviews as necessary

5. Ethics approval (if required)

•Joint agreement on ethics submission

•Response to reviews as necessary

6. Conduct of research

•Regular reports back to research and evaluation advisory board, as previously agreed

7. Reporting research

• Writing of reports and papers, as previously agreed

• Publication of results, as previously agreed

• Communication of results - to staff, shareholders and stakeholders, policy makers, media, professional groups as appropriate

8. Using the research

•Analysis of implication of research results for Healthdirect programs

• Incorporation of findings into program development and operations as appropriate


Part 6: Communications

Purpose

To improve communications across the organisation and to external agencies, an important

part of this Strategy is to adopt processes that ensure research and evaluation results are

made available in an effective and timely manner that will:

Drive knowledge transfer inside and outside Healthdirect

Strengthen Healthdirect’s profile as an authoritative voice on future directions and

investment priorities in telehealth and online health services

Boost public confidence in the effectiveness, safety and value of Healthdirect’s

services.

Aim

Increase awareness of Healthdirect research across the Company, stakeholders and

shareholders and to the wider community.

What we will do

Every research project and evaluation program will have an accompanying communications

plan. This will consider when and how results will be provided to the following groups:

Healthdirect board, program managers and staff

Shareholders and stakeholders

Research partners and potential research partners

Policy makers

Users of Healthdirect services and general public.

This will include:

Developing a page on Healthdirect intranet “Pulse” to provide updates for staff

Providing updates for Committees part of the Healthdirect governance structures

Refine the process for publishing and presenting research and evaluation findings to

a range of audiences. Providing opportunities for all Healthdirect staff to participate in

this process.


Appendix 1

Levels of Monitoring and Evaluation

Type Description Example Questions to be answered

Monitoring

/ Routine

data

collection

Provides quantitative information about

services. Needs further analysis and

interpretation.

How many people access a service?

Wait times for service access?

How much money was spent on the

service?

Formative Formative evaluation provides information

on:

How new services might be developed

How existing services might be improved

What is the problem to be solved?

Where is the problem and how big or

serious is it?

What are the characteristics and needs of

the target population?

What is the most appropriate solution?

What is the expected outcome and how to

measure if this is achieved?

Process Process evaluation looks at how a service

is delivered. It can help differentiate

ineffective services from failures of

implementation.

For ongoing services it can be used for

continual improvement.

How is the service being implemented?

Are the activities being delivered as

intended?

Are participants being reached as

intended?

Summative Summative evaluation generally reports

when a service has been running long

enough to produce results. It may also be

known as:

Outcome evaluation – did the service

cause demonstrable effects on the agreed

outcomes?

Cost effectiveness – addresses the

questions of efficiency.

What are the net effects?

To what extent can changes be attributed

to the service?

Is the service the best use of the

resources?


Appendix 2

Evaluation and monitoring framework - Healthdirect services and programs

Key drivers

Levels of

monitoring

and evaluation

Demand Targeting Effectiveness Quality and

safety

Awareness Equity Cost-

effectiveness /

value

Impacts on

health

services

Client

satisfaction

Investigating the needs and expectations of target populations for Healthdirect services.

Investigating whether Healthdirect services are reaching the right people and meeting their needs.

Asking about whether Healthdirect is achieving its intended goals.

Benchmarking Healthdirect services against standards and expectations of quality and safety.

Ensuring Healthdirect services are known in the community and used appropriately

Ensuring vulnerable populations have equality of access to and care from Healthdirect’s services.

Measuring the value of Healthdirect services.

Investigating wider impacts of Healthdirect services and programs.

Evaluating the values and opinions of Healthdirect service users.

Input/output

monitoring

Data- driven

monitoring

(usually done

routinely) of key

inputs and

outputs of a

program.

Who is using the service/program?

How are they using it?

What is the frequency of use of the program/service? What are the main reasons for use? What would the user otherwise have done?

Are services provided in accordance with agreed guidelines and standards?

Have there been adverse events?

Which users of services are from populations groups of interest? e.g. Aboriginal and Torres Strait Islander people; CALD; people with disabilities (especially those who are blind and/or deaf); people with low educational levels

What are the direct costs of implementing the service?

Did the user of the service intend to use a different health provider?

Client satisfaction surveys.

Process

Evaluation

Examines how a

program is

delivered seeking

potential

improvements.

Has the program been implemented in a way intended? Has technology been utilised optimally?

What factors determine whether the advice given is followed?

What are the barriers and facilitators to delivering the service as intended?

What caused any adverse events?

How does the public find out about services? How are services marketed to the public?

How well is the program or service designed to accommodate the needs of at-need population groups? Barriers and facilitators to implementing these design elements?

What are the real and hidden costs of service implementation? What costs are higher or lower than expected?

What aspects of Healthdirect services determine the use (appropriate and otherwise) of other health services?

Do front line service deliverers follow courtesy guidelines? Are these culturally sensitive?

What are the barriers and facilitators to this?

Formative Examines how a What is the unmet What are the What should be How can the What is the level What barriers What outcomes What potential What is an


Evaluation program can be

developed and

optimised in a

specific context.

need for

services? How

does this differ in

population

subgroups?

appropriate population cohorts for each service?

the expected outcomes for users of Healthdirect services?

quality and safety of the service be improved?

Are adverse events handled appropriately?

of awareness of Healthdirect services among key population groups?

hinder access by various population sub-groups?

are direct relevance to cost-effectiveness? Where can costs be reduced?

impacts on other health services are of interest?

acceptable level of client satisfaction? How is this best measured?

Summative

Evaluation

Examines net

effects of a

program.

What is the

demand for the

service being

evaluated? Is that

demand being

met?

What are the barriers to effective use of current services? eg. English language, health literacy, technology access, other variables.

What are the actual outcomes for users of services and how do they differ from those of non-users?

To what extent do people who use services follow the advice given?

Do services ease after-hours pressures on GPs?

Do services improve health literacy?

Do people who use services interpret the advice they are given accurately?

Are services responsive to disabilities, cultural and language needs? Is this demonstrated in the use of data?

What is the public opinion of services, amongst key population groups?

What is their level of satisfaction with services?

What is likelihood of using / recommending services?

Ares services seen as culturally safe?

Do all population subgroups have adequate access to services?

What are the costs of services benchmarked against those for similar services in Australia and internationally?

What return on investment does the program or service provide?

What value do Australians / users place on services? Is this commensurate with actual operational costs?

Is there evidence that use of services changes the use of other health / healthcare services and what are the cost implications of this?

What are the net effects of the service or program on other health services?

To what extent can changes be attributed to Healthdirect services?

What is the level of satisfaction with services?


Appendix 3

Key drivers for evaluation

1. Demand

What is the demand for Healthdirect services?

Demographics of this demand

What is the unmet need for Healthdirect services?

Demographics of this.

2. Targeting

Are Healthdirect services targeting the right population cohorts?

Does each Healthdirect service reach the designated population cohort/s?

What are the barriers to effective use of current Healthdirect services?

What impact does English language literacy have on use of Healthdirect services?

To what extent does technology influence targeting of Healthdirect services?

What impact does technology literacy have on use of Healthdirect services?

3. Effectiveness

What are the expected outcomes for users of Healthdirect services?

What are the actual outcomes for users of Healthdirect services and how do they differ from

those of non-users?

To what extent do people who use Healthdirect services follow the advice given?

What factors determine whether advice given is followed?

Are users of Healthdirect services more or less likely to:

o Receive after-hours treatment / care?

o Visit an Emergency Department for treatment?

Do Healthdirect services ease after-hours pressures on GPs?

Do Healthdirect services improve health literacy?

4. Safety and appropriateness

Are Healthdirect services provided in accordance with agreed guidelines and standards?

Do people who use Healthdirect services interpret the advice they are given accurately?

What are the barriers to compliance with Healthdirect advice?

Are Healthdirect services responsive to disabilities, cultural and language needs? Is this

demonstrated in the use data?

Is there evidence for adverse events for people who use Healthdirect services?

Are adverse events handled appropriately?

5. Awareness

What is the level of awareness of Healthdirect services among:

o General population

o Young mothers

o People with chronic illness and their carers

o Staff in residential care facilities

o Community nurses

o GPs

o Pharmacists

What is their opinion of Healthdirect services?

What is their level of satisfaction with Healthdirect services?

What is likelihood of using / recommending Healthdirect services?


How do people learn about Healthdirect services?

6. Equity

What do the data say about use of Healthdirect services by:

o Aboriginal and Torres Strait Islander people

o People whose first language is not English

o People with disabilities (especially those who are blind and / or deaf)

o People with low educational levels

What barriers hinder access by these groups?

Are Healthdirect services seen as culturally safe by Aboriginal and Torres Strait Islander

people?

7. Cost effectiveness / value / cost-shifting

Costs of Healthdirect services can be benchmarked against those for similar services in

Australia and internationally.

What value do Australians / users place on Healthdirect services? Is this commensurate with

actual operational costs?

Is there evidence that use of Healthdirect services changes the use of other health /

healthcare services and what are the cost implications of this?

8. Impacts on health services

What health service/s does this service supplement or substitute for?

What services are likely to be affected by this service (positively and negatively)?

Are there impacts on health workforce?

How will unintended consequences be assessed?

9. Client satisfaction

Did people get what they wanted / needed from the service or service?

Were their expectations reasonable?

If someone else used the service on behalf of a patient, was the patient satisfied?

Did client understand and act appropriately on the advice given?

Would clients use the service again?

Would clients recommend the service to others?

Are procedures to address clients’ grievances effective? How is this effectiveness measured?

healthdirect australia research and evaluation …...research and evaluation strategy aligns to...

Documents