assessment of ict investment in healthcare: an overview › staff › e.w.berghout ›...

Evaluation of ICT Investment in Healthcare: Insights and Agenda for Future Research

Arviansyah, Egon Berghout, Chee-Wee Tan

University of Groningen, Groningen, The Netherlands

[email protected]

[email protected]

[email protected]

Abstract: The recent downturn in global economy exerts mounting pressure on the justification of ICT budgets within organizations. Effective evaluation of ICT investments is therefore deterministic of organizations’ ability to maximize the business value to be extracted from such investments. The same can be said for healthcare institutions. Increasingly, healthcare institutions are pressured to optimize ICT investments for improving patient care. Yet, extant literature is fragmented on the focal concepts to be utilized for ICT evaluations in healthcare. By conducting a systematic literature review of 99 scholarly articles that have been published between 2003 and 2008, this study endeavors to shed light on contemporary theoretical and methodological trends that are pertinent to ICT evaluations in healthcare.

Based on our literature review, three main findings are noted. First, we detect a gradual progression of ICT evaluation methods from the conventional standard of random controlled trial (RCT) to mixed-methods and triangulation. This, in turn, translates to a greater diversity of constructs being utilized for ICT evaluation and an expansion in operationalization techniques for measurement purposes.

Second, we identified the five most important stakeholders and evaluative dimensions to be included in the ICT evaluation process. Together, these identified stakeholders and evaluative dimensions represent core elements to be incorporated into ICT evaluations in healthcare such that evaluations can be undertaken in an informative and purposeful fashion.

Third, we identified several distinctive contextual characteristics of the healthcare sector that deserve further research. These characteristics include the multi-stakeholder environment, the industry’s emphasis on safety as well as the knowledge intensive nature and handcrafted type of work being performed.

This study enriches extant literature by synthesizing findings from past studies to glean insights into relevant concepts for ICT evaluation in healthcare and propose an agenda for future research along this direction.

Keywords: information and communication technology evaluation, information technology evaluation, information technology investment, information technology in healthcare, systematic literature review.

1. Introduction

Anderson & Aydin (2005) reported a 50% failure rate for healthcare information systems due to user-resistance and staff interference problems. They argued that ICT failures in healthcare are caused by overemphasizing technological and economic aspects of ICT investments with little regard for organisational and socio-political issues. Sistrom (2005) argued that despite few tangible benefits and significant adverse effects on patients and the medical profession, the healthcare sector has invested heavily in information technology.

The healthcare sector is slow in adopting technology (Kaplan, 1997, Christensen and Remler, 2009, Sistrom, 2005). Kaplan (1997), who claimed that the lag of the healthcare sector in technology adoption, is caused by a lack of knowledge about the cultural and socio-political aspects of ICT investments. For this reason, Friedman & Wyatt (2006) predicted that there will be a demand for

mailto:[email protected]



evidence-based technology to facilitate adoption decisions by healthcare organisation given their emphasis on safety.

Arguably, two reasons justify the urgency of effective ICT evaluation in the healthcare sector. One, effective evaluation of ICT investments provides a means of drawing comparisons among competing project alternatives such that investment decisions can be undertaken in an informative and purposeful manner (Farbey et al., 1993). Two, substantial challenges exist with regards to the effective evaluation of ICT investments in the healthcare sector due to its contextual uniqueness and complexity (Braaksma et al., 2006)

By conducting a systematic review of extant literature, this study thus provides an overview of contemporary theoretical and methodological trends in the evaluation of ICT investments within the healthcare sector.

2. Theoretical foundation

2.1. Evaluation of ICT in healthcare

Ammenwerth (2004) defines evaluation as the act of measuring or exploring properties of a health information system that serves to inform decision-making about the system. That is, evaluation represents the process of gauging resources for their merit and worth through comparing alternatives and ending with the act of choice (Guba and Lincoln, 1981, Berghout, 1997).

The basic types of evaluation methods can be divided into formative-summative type and objectivist-subjectivist method (Friedman and Wyatt, 2006). The formative type of evaluation is conducted during the life time of a project and intended to provide input for development and design. Conversely, summative is conducted at the end of a project and intended to look for system’s achievement (Lærum, 2004, Stoop, 2005). Whereas objectivist methods deal primarily with quantitative measurements, subjectivist methods deal with qualitative measurements.

Goodman et al., (1999) argued that technologies may be assessed at different stages of diffusion and maturity. Therefore, in order to meet the needs of various policy makers seeking information throughout technological lifecycles, Goodman et al. (1999) suggested an iterative process of healthcare technology evaluation by taking into consideration its rapid change and development.

For evaluation to be comprehensive, Remenyi et al. (2007) argued that evaluation should incorporate the opinions of all major stakeholders and focus on a full range of benefits or outcomes, both tangible and intangible. They argued that while there is no self-evident methodology for cost-benefit analysis of ICT, organisations still need to be able to evaluate their business process changes and the accompanying ICT investments.

To capture the benefits of new technologies, the utilization of robust methods, with appropriate evaluative dimensions that take into account indirect benefits and costs, is needed to enhance decision-making within healthcare organizations (Sorenson et al., 2008). Conceivably, more knowledge on appropriate stakeholders and evaluative dimension are required from a value-centric perspective.

3. Method

The literature review is conducted by examining published research from EBSCO Host Research Database, which includes Business Resource Premier – a database of developments in the field of international business and management and MEDLINE – a database of developments in the field of medical information, nursing and healthcare systems.

The search strategy and flow of systematic review is shown in details in Figure 1. Several restrictions are applied in the phase of detail reviewed to establish the scope and framework of the research. Table 1 shows the definition of each term and the criteria used for article selection to comprise the preceding definitions. All the articles are analysed and synthesised according to their type of study, evaluation method, stakeholders and evaluative dimension.

Table 1: Definition and criteria

Term Definition References Criteria

Evaluation Process of describing the realisation of resources for their merit and worth through judging and comparing alternatives to other comparable entities and/or with a set of standards suitable for its context, ended with the act of choice in order to improve the design or the implementation of resources

(Ammenwerth et al., 2004, Berghout, 1997, Guba and Lincoln, 1981, Remenyi et al., 2007)

Articles describe evaluation process using specific evaluative dimensions.

ICT Terminology derived from the term of Information Technology with a high dependency on data communication. Known also as a set of information processing, storage and transmission technologies emerging from microprocessor advancement

(Cantwell et al., 2004, Remenyi et al., 2007)

Articles describe the involvement of ICT as the chain in delivering value, i.e. supporting organisational processes. Computer-based training or any kinds of education related matters are excluded as these systems are not part of direct patient care (Ammenwerth and de Keizer, 2005)

Investment Use of money to get a profit or to increase the probability of a successful a business activity over a period of time. To be considered as investment, the transformation of capital into assets should last for at least a period of more than one year; allowing for an evaluation of its worthiness and its alignment with overall strategy

Longman Dictionary of Contemporary English

(Berghout, 1997, Hogbin and Thomas, 1994)

Articles describe the consequences of investment decision and state the outcome of the investment based on the evaluation process. The worthiness based on the evaluative dimensions could be tangible or intangible or a combination of them.

Healthcare Prevention, treatment, and management of illness and the preservation of mental and physical well-being through the services offered by the medical and allied health professions.

The American Heritage Dictionary of the English Language

Articles describe the evaluation of ICT in the healthcare context.

Keywords

EBSCO Host

Research

Databases

MEDLINEBusiness

Resource Premier

healthcare OR

medical OR

hospital OR health

care

investment OR cost

OR benefit* OR

economic* OR cost-

benefit analysis

ict OR information system* OR

information technology OR

communication technology OR

informatics OR computer*

assessment OR

evaluation OR

valuationAND AND AND

Limiters set

in: AB - Abstract in: no field (none)

Starting year 2003

BRP: Scholarly (Peer Reviewed) Journals - Publication Type: Academic Journal

MEDLINE: English Language - Publication Type: Journal Article - Abstract Available

Sort: relevance

Result: 291

Abstract reviewed

Full text

availability

Articles reviewed

in detailBackward searchForward search

Reference &

Author

Reference &

Author

Excluded: 54

Articles included: 99

Result: 212

Result: 109

Result: 65 Addition: 44

BRP: 31

MEDLINE: 260

BRP: 26

MEDLINE: 186

BRP: 21

MEDLINE: 88

Applying restrictions/criteria/

qualifications

Duplication: 1

Duplication: 2

10 articles catagorised as

education

1st phase

2nd

phase

3rd

phase

Impact factor: Journal Citation Report

Number of citation: Google Scholar

Figure 1: Search strategy for systematic review

4. Results

In this section we will describe the composition of the remaining articles from the systematic literature review along with the research’s result of stakeholders and evaluative dimensions.

Descriptive statistics from our literature review show that 46% of the articles belong to the category of

case or empirical studies, 30% to that of informational studies and 24% to that of review studies

(Figure 2). In the category of case or empirical study, most of the articles use combination methods or

multi approaches in their evaluation process (Figure 3). More researchers use quantitative methods

than qualitative studies during the period of published-articles (Figure 3)

Figure 2: Type of study from the remaining articles

Figure 3: Evaluation method from the category of case and empirical study

46%

30%

24% Case/empirical

Informational

Review

26%

33%

41% Qualitative

Quantittative

Mixed/multi

71

4.1. Analysis of stakeholders

Many stakeholders were identified throughout the study. 17 categories were identified and their definitions summarised. Sorted by their frequency, number of citation and impact factor (Figure 4), these stakeholders are: Physician, Nurse, Patient, Healthcare organisation, Clinician, (End) User, IT/IS department/developer related, Staff, Assistant related position, Client, Head of clinic/laboratory manager, Clinical researcher/hospital scientist, Top management, General Practitioner (GP)/Doctor, Medical student, Head of division/manager, Community/external.

Figure 4: Stakeholders stacked bar

0 5 10 15 20 25 30 35 40

Community/external

Head of division/manager

Medical student

General Practitioner (GP) /Doctor

Top management

Clinical researcher/hospital scientist

Head of clinic/laboratory manager

Client

Assistant related position

Staff

IT/IS department/developer related

User - undefined

Clinician

Healthcare organisation

Patient

Nurse

Physician

Frequency Absolute no. of cited Relative no. of cited Impact factor

Figure 5: Stakeholders stacked bar without frequency

Figure 6: Difference between stacked bar with and without frequency (stakeholders)

0 2 4 6 8 10 12 14 16 18

Community/external


Medical student




Client


Top management


Staff

Clinician

User - undefined


Patient

Physician

Nurse

Absolute no. of cited Relative no. of cited Impact factor

0 2 4 6 8 10 12 14 16 18 20

Top management

Community/external

Client


Staff


Medical student



User - undefined




Clinician

Nurse

Patient

Physician

Difference

Figure 4 and Figure 5 compare contribution of number of citations, impact factor and frequency across the

different categories of stakeholders. For comparison of these figures, Figure 6 illustrates the bar chart of score difference from these two stacked bars across every category of stakeholders. All bar charts are sorted in descending order. Figure 4 illustrates the importance of stakeholders on the basis of impact factor, relative and absolute number of citation and frequency. From the graph’s rank order, we can see that Physician, Nurse, Patient, Healthcare organisation and Clinician are the top five most important stakeholders.

Furthermore, Figure 5 illustrates the importance of stakeholders on the basis of impact factor and relative and absolute number of citation. We can see that Nurse, Physician, Patient, Healthcare organisation and User are the top five most important stakeholders. Since these two graphs are the result of a weighted average

calculation; the importance of these 17 stakeholders can be inferred from Figure 4 and Figure 5.

In addition, Figure 6 illustrates the difference between those two analyses of importance. From Figure 6, it is clear that Physician, Patient, Nurse, Clinician and IT/IS department or developer are the top five stakeholders which have large differences between the two analyses of importance. Interestingly, IT/IS department and developer emerge as stakeholders with large difference (Figure 6) despite not emerging as the top five of

important stakeholders in both importance analysis (Figure 4 and Figure 5).

4.2. Analysis of the evaluative dimensions

Many evaluative dimensions were revealed throughout the study. 28 categories were identified their definitions summarised. Sorted by their importance (Figure 7), these evaluative dimensions are: Usage, Satisfaction, Time related, Usability, Data or information quality, Organisation, Human, Technology, Medical outcome, Service quality/quality of care, Cost related, Performance, Quality of life (QOL), Functionality, Efficiency, Technical specification/requirement, Acceptance or attitude, Accurateness, Guidelines accordance, Effectiveness, Automation, Reliability, Knowledge related, Usefulness, Operational, External, Strategic, and Managerial.

Figure 7: Evaluative dimensions stacked bar

0 5 10 15 20 25 30 35 40

Managerial

Strategic

External

Operational

Usefulness

Knowledge related

Reliability

Automation

Effectiveness

Guidelines accordance

Accurateness

Acceptance/attitude

Technical specification/requirement

Efficiency

Functionality

QOL

Performance

Cost related

Service quality/quality of care

Medical outcome - desired/undesired

Technology

Human

Organisation

Data/information quality

Usability

Time related

Satisfaction

Usage

Frequency Absolute no. of cited Relative no. of cited Impact factor

Figure 8: Evaluative dimensions stacked bar without frequency

Figure 9: Difference between stacked bar with and without frequency (evaluative dimensions)

0 5 10 15 20 25 30

External

Managerial

Strategic

Operational

Effectiveness

Knowledge related

Reliability

Usefulness

Cost related

Accurateness

Acceptance/attitude


Automation



Efficiency

Organisation

Usability

Technology

Functionality


Performance

Satisfaction

Human

QOL

Time related


Usage

Absolute no. of cited Relative no. of cited Impact factor

0 2 4 6 8 10 12 14 16 18

Automation

Managerial

QOL

Strategic

Usefulness

Operational

External

Knowledge related

Performance

Reliability

Functionality


Human




Efficiency

Accurateness

Effectiveness

Usage

Technology

Acceptance/attitude

Time related

Satisfaction

Organisation


Usability

Cost related

Difference

Figure 7 and Figure 8 compare contribution of number of citation, impact factor and frequency across every

category of evaluative dimensions. For comparison of these figures, Figure 9 illustrates the bar chart of score difference from these two stacked bars across every category of evaluative dimensions. All bar charts are sorted in descending order. Figure 7 shows the importance of evaluative dimensions based on impact factor, relative and absolute number of citation and frequency. From the graph’s rank order, we can see that Usage, Satisfaction, Time related, Usability and Data/information quality are the top five most important evaluative dimensions.

Furthermore, Figure 8 illustrates the importance of evaluative dimensions on the basis of impact factor and relative and absolute number of citation. We can see that Usage, Data/information quality, Time related, Quality of life (QOL) and Human are the top five most important evaluative dimensions. Since these two graphs are the result of a weighted average calculation, the relative importance of these 28 evaluative

dimensions can be inferred from Figure 7and Figure 8.

In addition, Figure 9 illustrates the difference between those two analyses of importance. From Figure 9, we can see that Cost related, Usability, Medical outcome, Organisation and Satisfaction are the top five evaluative dimensions which have large difference between the two analyses of importance. Interestingly, Cost related and Medical outcome emerge as evaluative dimensions with large difference (Figure 9) despite

not emerging as the top five of important evaluative dimensions in both importance analysis (Figure 7 and

Figure 8).

5. Discussion

In this section we will discuss findings of the stakeholders and evaluative dimensions revealed by our systematic literature review.

In general, there is insufficient scholarly attention being paid to the derivation of guidelines to assist stakeholders in evaluating ICT investments. The guidelines that consist of detailed study protocol of all the decisions and steps might provide benchmark for other researcher in conducting further evaluation. These guidelines are beneficial for allowing others to learn and to develop the framework as well as to test the robustness of its applicability (Ammenwerth et al., 2003).

5.1. Progress of ICT evaluation in healthcare: construct diversity and methodological pluralism

Evaluation method of ICT investment in the healthcare context is influenced by conventional medical research. The golden standard for assessing medical interventions is the Randomised Controlled Trial (RCT) which is also used in evaluating ICT investment in healthcare sector (Berg, 2004). Examples include the studies of Apkon et al. (2005), Fung (2006) and May, Mort, Williams, Mair, & Gask (2003). Quantitative evaluation methods have been dominant for a period of 20 years (Ammenwerth and de Keizer, 2005). Yet, in spite of its prevalence, the RCT design is unsuitable for explaining why and how certain ICT is being used (Stoop, 2005). Isolating information system’s component and its social interaction, likewise introducing new medical intervention or new medicine, is dubious, labour intensive and expensive method (Berg, 2004). The earlier studies of evaluation in medical informatics tended to lean towards the use of quantitative method and measurement.

Approaches that analyse qualitative data are needed to measure intangible benefits and explain social phenomena. In later development, there is a greater consideration for the social and organisational aspects in evaluation studies and to incorporate specific measures such as technological, human and organisational issue in the evaluation process (Yusof et al., 2008, Oroviogoicoechea et al., 2008, de Keizer and Ammenwerth, 2008). In spite of this urgency, the quality of RCT’s publications was significantly higher than the quality of non-RCT studies which comprise 70% of the overall studies (de Keizer and Ammenwerth, 2008). Moreover, review made by Rahimi & Vimarlund (2007) showed that despite the recent dominance of economic and organisational aspects in the evaluation of health information system, we are still missing a standard framework for evaluating effects and outputs of implementation and use of ICT in the healthcare context.

Furthermore, various notions of value emerge from the research. Benefit is the notion value used mostly in the articles. Benefit refers to all positive consequences of an IS investment as the antonym of sacrifices (Renkema and Berghout, 1997). These various notions of value imply that there are many perspectives

available to the researcher in evaluating ICT investments within the healthcare setting. The progression of evaluation method shows a shifting from using single quantitative or qualitative method to a combination approaches or triangulation to encompass more comprehensive output. Recent studies have begun to employ diversity of constructs and measurement techniques to meet the challenges of RCT method, such as using combination of method or multi approaches in their evaluation process.

5.2. Stakeholders in healthcare settings: involvement and importance

From Figure 4, the analysis shows that Physician, Nurse, Patient, Healthcare organisation and User are the top five most important stakeholders. In addition, Clinician and User have somewhat similar ranking based on

Figure 4 and Figure 5. Physician and Nurse, as part of healthcare team, have an influential role to play in

healthcare spending decisions. It is invaluable to understand their involvement as subjects conducting medical practice and as users utilizing specific ICT for medical treatment or medical administration.

Inevitably, patients, as one of the important stakeholders is the object of medical treatment as well as the subject or ‘customer’ of health organisation, also take as the user of technology in certain healthcare setting. Health organisation as important stakeholder means that the management which is providing and delivering service to ‘customer’ (patient and client) should be involved in assessing the worthiness of ICT investments. The stakeholders that emerged based on these two analyses are physician, nurse, patient and healthcare organisation. These four stakeholders should be highly involved in evaluating ICT investment in the healthcare setting.

However, these four stakeholders are not necessarily involved in the ICT evaluation process for specific types of technologies and healthcare settings (Cariello, 2003, Fung, 2006, Kaur et al., 2004, Kilbridge et al., 2006, Lorence and Greenberg, 2006, Ramnarayan et al., 2006, Thompson et al., 2003, Zheng et al., 2005, Beuscart-Zephir et al., 2005, Harison and Berghout, 2006). For example, evaluation of computerised templates in primary care only involved clinician whereas the comparison of paper-based form with computer-based evaluation only involved the tele-rehabilitation coordinator and the research assistant.

Interestingly, despite a general belief that IT/IS department should be involved in evaluation of ICT investment, IT/IS department appears to be rarely consulted.

5.3. Evaluative dimensions of ICT Investment in healthcare

From Figure 7, the analysis shows that Usage, Satisfaction, Time related, Usability are four most important evaluative dimensions. Data or information quality and organisation have a similar rank. Usage, usability and user satisfaction are related each other in showing the worthiness of ICT investments. System with higher usage usually implies its usability and satisfaction.

High vigilance coupled with strong pressure in the healthcare sector demands fast and accurate information. Consequently, time related, data/information quality and quality of life emerged as important evaluative dimensions. Timed related dimension is important since most activity and process in the healthcare sector require agility; ICT should help in achieving better time compare to the old process. Data/information quality dimension is important since problem that relates to life or death demands accurateness and reliable information as the basis for deciding medical treatment. Through this medical treatment, ICT should have significant role in increasing the patient’s Quality of life (QOL).

The graphs show that Satisfaction, Usability and Organisation have lower importance when frequency is omitted from the analysis (Figure 8). Conversely, Quality of life (QOL) and Human category emerge as important evaluative dimensions (Figure 8). From the difference analysis (Figure 9), it seems that many researchers used Medical outcome and Cost related category to evaluate ICT investments; however, these two dimensions get lower ranking of importance when frequency is omitted from the analysis. Medical outcome is a well-establish evaluative dimension to evaluate medical intervention such as the new introduction of new drugs or new medical treatment. It is possible that this phenomenon is caused by the progression of evaluation method. Likewise, the cost related as evaluative dimension is frequently mentioned in many researches. A possible explanation to this phenomenon is that the importance of cost is highly recognised; however, specific cost such as adverse drug event (ADE) is more difficult to measure. Moreover, there is also a possibility that many healthcare organisations are government funded institutions.

Organisation, human, and technology perspective emerged as important dimensions which show the potential direction of evaluating ICT in healthcare. The complete set of evaluative dimensions for general type of ICT and healthcare setting are Usage, Satisfaction, Time related, Usability, Data or information quality, Organisation, Quality of life (QOL), Human and Technology.

5.4. Distinctive contextual characteristics of the healthcare sector

The healthcare service is highly related to the well being of patients as the customers, which make the outcome is less predictable. Patient’s safety should then be the foremost concern for healthcare organizations. Patient’s safety is accomplished by both minimising errors and adverse effects and enhancing medication processes.

Sistrom (2005) described healthcare as a labour-intensive sector where by the work process is more cognitive than physical. This process, which involves patients together with patient families, medical professions and administrative personnel, creates a complex, a highly structured and quite a heterogeneous culture. Therefore, Sistrom (2005) argued that ICT tends to disrupt existing social structure as well as established clinical practice and core work process by reengineering the cognitive process and the social structure of the healthcare profession. This problem will lead to dissatisfaction and resistance. An example of resistance occurring in a university medical centre was presented by Lorenzi et al. (1997). The new technology encountered challenges as it disturbed and forced changes to established processes and practice routines. That is, medical staff sees the technology as the enforcement of will and value by the management.

The healthcare sector is considered a complex and a multi-stakeholder environment with a split decision-making organisation. Lorenzi et al. (1997) mentioned several reasons why healthcare is a complex service organisation. Based on Mintzberg’s type of organisations, they argued that the healthcare organisation is a combination of a federal structure (i.e. homogenous with regional unit), a machine bureaucracy (i.e. top formal process controlled), a professional bureaucracy (i.e. autonomously professional activities with professional value and culture controlled), and a networked organisation (i.e. independent individuals and sub-organisations that link each other for a certain objective or task) (see also Southon et al, 1999).

The healthcare sector is also encouraged to share knowledge due to the knowledge intensive characteristic of the organisation. The knowledge sharing in a knowledge intensive environment will support better decision making among medical professions in one organisation and between organisations in the healthcare sector. Evaluating healthcare information system through an enterprise perspective will encounter difficulties in this multi-stakeholder environment (Connell and Young, 2007).

6. Conclusions and recommendations

A gradual progression from conventional standard of RCT to mixed-methods and triangulation indicates a trend and growing demand for multidisciplinary ICT evaluation frameworks. There is a challenge to build more robust and comprehensive evaluation process. One alternative way is by incorporating a multidisciplinary team of evaluator which might reveal diverse insight as well as enrich the understanding of healthcare contextual situation.

Determination of success and failure in ICT investment project would depend on how stakeholders define their needs and perceptions of the technology and how evaluator elicits them by using the evaluative dimensions. Therefore, evaluator should clearly identify the key stakeholders as well as utilize proper evaluative dimensions. The identified stakeholders and evaluative dimensions in this study represent the core elements to be incorporated into ICT evaluations in healthcare such that evaluations can be undertaken in an informative and purposeful fashion.

Furthermore, as the distinctive contextual characteristics of the healthcare sector might influence organizations’ choice of evaluation constructs and measurement methods, future investigations are needed to decipher their impact on ICT evaluations in healthcare.

References

AMMENWERTH, E., BRENDER, J., NYKANEN, P., PROKOSCH, H.-U., RIGBY, M. & TALMON, J. 2004. Visions and strategies to improve evaluation of health information systems. Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. International Journal Of Medical Informatics, 73, 479-491.

AMMENWERTH, E. & DE KEIZER, N. 2005. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982-2002. Methods Of Information In Medicine, 44, 44-56.

AMMENWERTH, E., GRABER, S., HERRMANN, G., BURKLE, T. & KONIG, J. 2003. Evaluation of health information systems-problems and challenges. International Journal Of Medical Informatics, 71, 125-135.

ANDERSON, J. G. & AYDIN, C. E. 2005. Evaluating the organizational impact of healthcare information systems, Springer Verlag.

APKON, M., MATTERA, J. A., LIN, Z., HERRIN, J., BRADLEY, E. H., CARBONE, M., HOLMBOE, E. S., GROSS, C. P., SELTER, J. G., RICH, A. S. & KRUMHOLZ, H. M. 2005. A randomized outpatient trial of a decision-support information technology tool. Archives Of Internal Medicine, 165, 2388-2394.

BERG, M. 2004. Health Information Management: Integrating Information Technology in Health Care Work, Routledge.

BERGHOUT, E. W. 1997. Evaluation of information system proposals: Design of a decision support method. Doctoral dissertation Delft University of Technology, Delft University of Technology.

BEUSCART-ZEPHIR, M.-C., ANCEAUX, F. O., MENU, H., GUERLINGER, S., WATBLED, L. & EVRARD, F. 2005. User-centred, multidimensional assessment method of Clinical Information Systems: a case-study in anaesthesiology. International Journal Of Medical Informatics, 74, 179-189.

BRAAKSMA, J., COMMANDEUR, A. & BERGHOUT, E. 2006. The Business Case for (ICT) Investment Evaluation in Nonprofit Organisations. Proceedings of the 13th European Conference on Information Technology Evaluation.

CANTWELL, J., GAMBARDELLA, A. & GRANSTRAND, O. 2004. The Economics and Management of Technological Diversification, Routledge.

CARIELLO, F. P. 2003. Computerized telephone nurse triage. An evaluation of service quality and cost. The Journal Of Ambulatory Care Management, 26, 124-137.

CHRISTENSEN, M. C. & REMLER, D. 2009. Information and Communications Technology in U.S. Health Care: Why Is Adoption So Slow and Is Slower Better? Journal of Health Politics Policy and Law, 34, 1011-1034.

CONNELL, N. A. D. & YOUNG, T. P. 2007. Evaluating healthcare information systems through an "enterprise" perspective. Information & Management, 44, 433-440.

DE KEIZER, N. F. & AMMENWERTH, E. 2008. The quality of evidence in health informatics: How did the quality of healthcare IT evaluation publications develop from 1982 to 2005? International Journal of Medical Informatics, 77, 41-49.

FARBEY, B., LAND, F. & TARGETT, D. 1993. How to assess your IT investment: A study of methods and practice, Butterworth-Heinemann.

FRIEDMAN, C. P. & WYATT, J. C. 2006. Evaluation Methods in Medical Informatics, Springer. FUNG, C. H. 2006. Computerized condition-specific templates for improving care of geriatric syndromes in a

primary care setting. Journal Of General Internal Medicine: Official Journal Of The Society For Research And Education In Primary Care Internal Medicine, 21, 989-994.

GOODMAN, C. S. & AHN, R. 1999. Methodological approaches of health technology assessment. International Journal of Medical Informatics, 56, 97-105.

GUBA, E. G. & LINCOLN, Y. S. 1981. Effective evaluation, Jossey-Bass Inc Pub. HARISON, E. & BERGHOUT, E. 2006. Measuring the effects of information systems on the performance of

operating rooms (OR). Centre of IT Economics Research. Groningen, The Netherlands: University of Groningen.

HOGBIN, G. & THOMAS, D. V. 1994. Investing in information technology: managing the decision-making process, London; New York: McGraw-Hill.

KAPLAN, B. 1997. Addressing organizational issues into the evaluation of medical systems. Journal of the American Medical Informatics Association, 4, 94.

KAUR, K., FORDUCEY, P. G. & GLUECKAUF, R. L. 2004. Prototype database for telerehabilitation. Telemedicine Journal And E-Health: The Official Journal Of The American Telemedicine Association, 10, 213-222.

KILBRIDGE, P. M., WELEBOB, E. M. & CLASSEN, D. C. 2006. Development of the Leapfrog methodology for evaluating hospital implemented inpatient computerized physician order entry systems. Quality & Safety In Health Care, 15, 81-84.

LÆRUM, H. 2004. Evaluation of electronic medical records: A clinical task perspective. Doctoral, University of Science and Technology.

LORENCE, D. P. & GREENBERG, L. 2006. The zeitgeist of online health search. Implications for a consumer-centric health system. Journal Of General Internal Medicine: Official Journal Of The Society For Research And Education In Primary Care Internal Medicine, 21, 134-139.

LORENZI, N. M., RILEY, R. T., BLYTH, A. J. C., SOUTHON, G. & DIXON, B. J. 1997. Antecedents of the People and Organizational Aspects of Medical Informatics. Journal of the American Medical Informatics Association, 4, 79-93.

MAY, C., MORT, M., WILLIAMS, T., MAIR, F. & GASK, L. 2003. Health technology assessment in its local contexts: studies of telehealthcare. Social Science & Medicine (1982), 57, 697-710.

OROVIOGOICOECHEA, C., ELLIOTT, B. & WATSON, R. 2008. Review: evaluating information systems in nursing. Journal Of Clinical Nursing, 17, 567-575.

RAHIMI, B. & VIMARLUND, V. 2007. Methods to evaluate health information systems in healthcare settings: a literature review. Journal Of Medical Systems, 31, 397-432.

RAMNARAYAN, P., WINROW, A., COREN, M., NANDURI, V., BUCHDAHL, R., JACOBS, B., FISHER, H., TAYLOR, P. M., WYATT, J. C. & BRITTO, J. 2006. Diagnostic omission errors in acute paediatric practice: impact of a reminder system on decision-making. BMC Medical Informatics And Decision Making, 6, 37-37.

REMENYI, D., MONEY, A. H. & BANNISTER, F. 2007. The effective measurement and management of ICT costs and benefits, Oxford; Burlington, MA, CIMA.

RENKEMA, T. J. W. & BERGHOUT, E. W. 1997. Methodologies for information systems investment evaluation at the proposal stage: a comparative review. Information and Software Technology, 39, 1-13.

SISTROM, C. 2005. The socioeconomic aspects of information technology for health care with emphasis on radiology. Academic Radiology, 12, 431-443.

SORENSON, C., DRUMMOND, M. & KANAVOS, P. 2008. Ensuring value for money in health care: the role of health technology assessment in the European Union. Copenhagen; European Observatory on Health Systems and Policies, 2008, 179 p. ilus (Observatory studies series, 11).

SOUTHON, G., SAUER, C. & DAMPNEY, K. 1999. Lessons from a failed information systems initiative: issues for complex organisations. International Journal of Medical Informatics, 55, 33-46.

STOOP, A. P. 2005. Evaluating ICT applications in health care: Studies from a sociotechnical perspective. Doctoral Dissertation, Erasmus Universiteit Rotterdam.

THOMPSON, D. S., OBERTEUFFER, R. & DORMAN, T. 2003. Sepsis alert and diagnostic system: integrating clinical systems to enhance study coordinator efficiency. Computers, Informatics, Nursing: CIN, 21, 22.

YUSOF, M. M., PAPAZAFEIROPOULOU, A., PAUL, R. J. & STERGIOULAS, L. K. 2008. Investigating evaluation frameworks for health information systems. International Journal Of Medical Informatics, 77, 377-385.

ZHENG, K., PADMAN, R., JOHNSON, M. P. & DIAMOND, H. S. 2005. Understanding technology adoption in clinical care: clinician adoption behavior of a point-of-care reminder system. International Journal Of

Medical Informatics, 74, 535-543.

assessment of ict investment in healthcare: an overview › staff › e.w.berghout ›...

Documents