Journal of Biomedical Informatics 42 (2009) 334–346

Contents lists available at ScienceDirect

Journal of Biomedical Informatics

journal homepage: www.elsevier .com/locate /y jb in

Creating and sharing clinical decision support content with Web 2.0: Issuesand examples

Adam Wright a,b,c,*, David W. Bates b,c,d, Blackford Middleton a,b,c, Tonya Hongsermeier a, Vipul Kashyap a,Sean M. Thomas e, Dean F. Sittig f,g

a Clinical Informatics Research & Development, Partners HealthCare System, 93 Worcester St., Wellesely, Boston, MA 02481, USAb Division of General Medicine, Brigham & Women’s Hospital, Boston, MA, USAc Department of Medicine, Harvard Medical School, Boston, MA, USAd Clinical and Quality Analysis, Partners HealthCare System, Boston, MA, USAe Epic Systems Corporation, Verona, WI, USAf UT—Memorial Hermann Center for Healthcare Quality and Safety, Houston, TX, USAg The University of Texas School of Health Information Sciences at Houston, TX, USA

a r t i c l e i n f o a b s t r a c t

Article history:Received 8 May 2008Available online 8 October 2008

Keywords:Decision support systems, clinicalDecision making, computer-assistedDecision support techniquesHospital information systemsMedical records systems, ComputerizedCooperative behavior

1532-0464/$ - see front matter � 2008 Elsevier Inc. Adoi:10.1016/j.jbi.2008.09.003

* Corresponding author. Address: Clinical InformatPartners HealthCare System, 93 Worcester St., WelleFax: +1 781 416 8912.

E-mail address: awright5@partners.org (A. Wright

Clinical decision support is a powerful tool for improving healthcare quality and patient safety. However,developing a comprehensive package of decision support interventions is costly and difficult. If used well,Web 2.0 methods may make it easier and less costly to develop decision support. Web 2.0 is characterizedby online communities, open sharing, interactivity and collaboration. Although most previous attemptsat sharing clinical decision support content have worked outside of the Web 2.0 framework, several ini-tiatives are beginning to use Web 2.0 to share and collaborate on decision support content. We presentcase studies of three efforts: the Clinfowiki, a world-accessible wiki for developing decision support con-tent; Partners HealthCare eRooms, web-based tools for developing decision support within a single orga-nization; and Epic Systems Corporation’s Community Library, a repository for sharing decision supportcontent for customers of a single clinical system vendor. We evaluate the potential of Web 2.0 technol-ogies to enable collaborative development and sharing of clinical decision support systems through thelens of three case studies; analyzing technical, legal and organizational issues for developers, consumersand organizers of clinical decision support content in Web 2.0. We believe the case for Web 2.0 as a toolfor collaborating on clinical decision support content appears strong, particularly for collaborative con-tent development within an organization.

� 2008 Elsevier Inc. All rights reserved.

1. Introduction [8] and Kawamoto in 2005 [11] have shown generally favorable

Clinical decision support systems are tools designed to help hu-mans make better clinical decisions. The most familiar types ofdecision support, such as drug–drug interaction alerts and preven-tive care reminders are targeted at physicians, but clinical decisionsupport systems can also be designed to influence the clinical deci-sion making of nurses, pharmacists, ancillary care providers, pa-tients and others involved in the process of decision making inclinical care. Substantial evidence suggests that clinical decisionsupport can improve the quality and safety of healthcare [1–15].Systematic reviews of the past two decades by Grimshaw in2006 [16], Johnston in 1994 [10], Hunt in 1998 [9], Garg in 2005

ll rights reserved.

ics Research & Development,sely, Boston, MA 02481, USA.

).

results, in the areas of diagnosis, therapy and prevention.Although the cumulative evidence that clinical decision support

is beneficial is compelling, many specific interventions have had noimpact, and adoption of advanced clinical decision support sys-tems has been limited to date. A number of factors have limitedadoption, including challenges with integrating decision supportinto workflow, uncertainty about medical knowledge, organiza-tional and socio-political challenges and limited adoption of thoseclinical information systems, such as Computerized Provider OrderEntry (CPOE) and Electronic Health Records (EHRs) which are usedas a vehicle for the delivery of clinical decision support interven-tions. But one of (and perhaps) the largest inhibitors of adoptionis that translating text-based medical knowledge into actionable,real-time clinical decision support is a Herculean task for any butthe largest hospitals, health systems and provider organizationsto take on alone. In fact, a recent systematic review [17] suggeststhat just four of the nation’s largest academic medical centersand integrated delivery networks have carried out the lion’s share

A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346 335

of studies in this area. The overall magnitude of the decision sup-port knowledge management problem is enormous—in the aggre-gate, it has been estimated the costs of knowledge management forEHRs alone in the United States is approximately $25 billion [18].

It seems plausible that the best approach for handling this hugetask is for people and entities to share and collaborate on thedevelopment of clinical decision support content [19]. Geographicdistance and competitive pressures have, to date, made such col-laboration difficult on a large scale. In this paper, we evaluate thepotential of Web 2.0 technologies to enable collaborative develop-ment of clinical decision support systems through the lens of threecase studies; analyzing technical, legal and organizational issuesfor developers, consumers and organizers of clinical decision sup-port content in Web 2.0.

2. Background

2.1. Earlier decision support sharing efforts

There have been a number of non-Web 2.0 efforts at sharingdecision support content to date. One of the earliest efforts at shar-ing clinical decision support content was the Arden Syntax MedicalLogic Module (MLM) repository. Arden Syntax is a standard forencoding event-driven rule based clinical knowledge for use inclinical information systems [20,21]. Knowledge modules encodedin Arden Syntax are known as MLMs. An MLM library, hosted atColumbia University, exists to facilitate the sharing of MLMs. Therepository can be accessed at http://www.dmi.columbia.edu/re-sources/arden/mlm/cpmc-mlm-index.html, and currently contains240 Arden-formatted MLMs.

Another significant knowledge sharing initiative was InterMed, acollaboration between Stanford, Harvard and Columbia [22,23].InterMed encoded clinical knowledge in a knowledge representa-tion formalism known as the Guideline Interchange Format (GLIF)[24–26] and piloted sharing these guidelines amongst the three par-ticipating sites. Likewise, the SAGE project [27], which included clin-ical partners Intermountain Healthcare, Stanford University, theMayo Clinic and the University of Nebraska as well as technologypartners GE Healthcare and Apelon terminology services has pilotedthe development and sharing of guidelines, including ones relatingto immunizations, diabetes and community-acquired pneumonia.

A particularly ambitious knowledge sharing initiative was theInstitute for Medical Knowledge Implementation (IMKI) [28]. IMKIwas a non-profit organization founded by Eclipsys, Epic, Siemensand GE, and was designed as a clearinghouse and repository forsharing a variety of clinical decision support content. Although IMKIwas extremely promising, it encountered funding and technical is-sues as well as issues with participants’ willingness to share deci-sion support content they had developed, and it dissolved in 2003.

At the time of their conception, each of these knowledge shar-ing efforts was greeted with significant enthusiasm, and it is clearthat great effort was expended in each of them. That said, none hasgained significant traction. Arden Syntax is implemented in onlythree commercially available clinical information systems[29,30], and portability of Arden-formatted MLMs has been limitedbecause of challenges relating to mapping concepts to the localvocabulary (referred to the in the literature as the ‘‘curly bracesproblem”) [31]. And, although InterMed was successful insofar asthe participating sites exchanged clinical guidelines, it did not leadto wide-spread sharing of clinical guidelines, or significant adop-tion of GLIF in commercially available clinical systems. And, as pre-viously mentioned, although IMKI was initially greeted with muchfanfare, it was relatively short-lived and did not result in signifi-cant sharing of decision support content.

It is worth noting that efforts to share decision support contentneed not be (and have not been) necessarily restricted to sharing

executable forms of content. For example, sharing a human read-able description of an alert or reminder (such as its inclusion andexclusion criteria, the interventions it suggests and the logic ituses) may, in many cases, be as useful as sharing a machine-read-able form, particularly since there are very few clinical informationsystems that can natively import and interpret machine-readabledecision support in any of the proposed standard formats. On theother hand, not all expressions of clinical knowledge qualify asdecision support. For the purposes of this paper, we limit clinicaldecision support to knowledge artifacts which are designed to de-liver some sort of real-time, point-of-care, clinical intervention(including an informational intervention) within a computerizedclinical information system. By this definition, a textual descriptionof an alert, reminder or order set would, for example, qualify; how-ever, a raw clinical practice guideline would not. Although suchguidelines contain specific clinical recommendations, they arenot generally designed to be implemented directly as interventionsinside of a clinical information system. That said, rules derivedfrom a guideline (or even a rule that displays portions of the guide-line in specific clinical contexts) would certainly qualify.

2.2. Introducing Web 2.0

The previously described efforts were mostly limited to a smallnumber of pre-approved participants and were not fully interac-tive. Web 2.0, by contrast, is characterized by online communities,open sharing, interactivity and collaboration [32]. Although Web2.0 is more of a movement or a philosophy than a precise technol-ogy, Web 2.0 applications share some common principles and pol-icies, namely:

1. Using the web as an application and content deployment plat-form. Flickr (http://www.flickr.com), an online photo site withmany Web 2.0 features, is an example of this characteristic.Long before Flickr, there were offline photo organizing and edit-ing applications. However, Flickr moved these tools to the webwhile keeping much of the richness of the offline applicationexperience. By moving the tools to the web, Flickr was also ableto provide capabilities such as online sharing and communitydiscussion that weren’t possible in offline applications.

2. Leveraging the web as a participatory and not merely as a pub-lishing platform. Some innovative web sites that exemplify thispractice are: (a) Wikipedia, where any user can add an entry tothe encyclopedia (http://www.wikipedia.org); (b) Del.icio.us(http://del.icio.us/) and Flickr, where users can automaticallycreate tags, annotate content and create folksonomies (ad hoctaxonomies that emerge from members of an online commu-nity applying tags to content) as opposed to centrally definedtaxonomies; (c) Collaborative spam filtering products thataggregate individual decisions of users related to what is andwhat is not spam [33].

3. Providing valuable content in addition to simply offering usefultools. This content does not necessarily have to be developed bythe same party that develops the tools themselves—instead, thecommunity plays a key role.

4. Treating users as co-developers. Real-time monitoring of userbehavior to see which new features are used, and how theyare used, thus becomes another required core competency.Websites like Flickr, deploy new builds every half hour [32]and features that are not used are removed, just as new featuresare added.

5. Supporting syndication of services and content, as opposed tocentral control. An example of this is Really Simple Syndication(RSS), which is a tool for syndicating content (such as news sto-ries or blog entries) so that it can appear on sites beyond thecontent author’s own (http://www.rssboard.org/rss-specifica-

336 A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346

tion). The Chicago Crime Map (http://chicagocrime.org/) isanother classic example of this principle. The map overlayscrime reports from the Chicago Police Department on a Googlemap, but was developed by a third party. Since both the policedepartment and Google provided services and content in anopen way, an unrelated party was able to harness bothresources to develop a new application.

Although apparently promising, Web 2.0 is not without contro-versy. In many Web 2.0 communities, a surprisingly small numberof people actually make valuable contributions. For example, theBoston Globe reports that, on Wikipedia, out of nearly 90,000 edi-tors, ‘‘0.7% of users, about 615 people, have made more than 50% ofedits”. And while the accuracy of Wikipedia is almost universallyregarded as surprising good given its lack of professional editorialcontrol, at least one independent evaluation found that it fell atleast somewhat short of a professionally edited encyclopedia[34], although the review pointed out that both encyclopediashas a disappointingly large number of errors.

Further, it is not entirely clear that Web 2.0 is actually differentfrom ‘‘Web 1.0”. Technically, Web 2.0 uses the same protocol (theHypertext Transfer Protocol, HTTP) and same markup format(HyperText Markup Language, HTML) as ‘‘Web 1.0”. Others have ar-gued that ‘‘Web 1.0” was as much designed around the idea of com-munity as Web 2.0. In an interview with IBM, Tim Berners-Lee, theinventor of the World Wide Web argued that ‘‘Web 1.0 was allabout connecting people. It was an interactive space, and I thinkWeb 2.0 is of course a piece of jargon, nobody even knows what itmeans. If Web 2.0 for you is blogs and wikis, then that is peopleto people. But that was what the Web was supposed to be all along.”[35]. Indeed, Amazon.com allowed users to provide feedback andratings of books in 1995 [36], and the first wiki software was devel-oped in 1994 [37] both long before the term Web 2.0 was coined.

On the other hand, the term Web 2.0 has sparked new interestin many fields, including healthcare. A panel entitled ‘‘Connections,Collaboration and Creativity: Exploring Web 2.0 Applications inHealth Informatics and Professional Development” was presentedat Medinfo 2007, the triennial international medical informaticsconference and a number of recent journal publications have ex-plored the uses and implications for Web 2.0 in medical informat-ics and medicine in general [38–42].

Because the concepts and technologies under the Web 2.0 um-brella are expansive and broad, it can be difficult to determinewhat is and what is not Web 2.0. Indeed, if one’s lens is too broad,it may quickly seem that the entire web fits under Web 2.0. Whilewe do not deign to conclusively define Web 2.0 in this paper, we doneed to define it for the purposes of this paper. In doing so, we fo-cus on two key aspects:

1. All Web 2.0 systems must be delivered over the web. Non-elec-tronic systems are excluded

2. All Web 2.0 systems must allow each user who can access thesystem the ability to participate in the discussion and develop-ment of content and must impose only a minimal amount ofauthority or editorial control. Tightly edited and controlledresources, such as eMedicine (http://www.emedicine.com/) orUpToDate (http://www.uptodate.com) are out of this scope.

1 The Informatics Review is an on-line serial devoted to helping clinicians andinformation systems professionals keep up with the rapidly changing field ofclinically-oriented, medical informatics. The site is owned and operated by Dean F.Sittig, Ph.D.

3. Case studies

Although most previous attempts at sharing clinical decisionsupport content have worked outside of the Web 2.0 framework,several initiatives are beginning to use Web 2.0 to share and col-laborate on decision support content. We present case studies ofthree efforts: the Clinfowiki, Partners HealthCare eRooms, and EpicSystems Corporation’s Community Library.

These three efforts were chosen because they represent threedifferent approaches to sharing of decision support: a world-acces-sible wiki for developing decision support content (the Clinfowiki),a collaborative web-based platform for developing decision sup-port within a single organization (Partners HealthCare eRooms)and an internet-accessible repository for sharing decision supportcontent among customers of a single clinical system vendor (EpicSystems Corporation’s Community Library).

These three efforts are not the only examples of Web 2.0 toolsand concepts being used for developing and sharing clinical deci-sion support content. For example, Partners HealthCare’s environ-ment is based on the Documentum eRoom product (EMCCorporation, Hopkinton, MA) which is used for similar purposesin other healthcare organizations including Intermountain Health-care. Likewise, Epic’s Community Library, built with Microsoft’swidely used Sharepoint software is just one example of severalsimilar forums hosted by major clinical information systems ven-dors, including GE [43], where customers can share content witheach other.

3.1. Case study 1: the clinfowiki

The Clinfowiki, (http://www.clinfowiki.org) short for ClinicalInformatics Wiki, was created on 27 July 2005 by Dean F. Sittig,Ph.D. The goal of this resource is to provide clinical informaticiansaround the world with a place to document and discuss many ofthe most important lessons they have learned in their day-to-dayinformatics-related activities. Over the course of the first severalmonths of operation, the editors of the Clinfowiki came to realizethat such a resource could also be used to share clinical decisionsupport content and created several sections on the Clinfowiki de-voted to various aspects of clinical decision support. The site cur-rently has an extensive section on CDS for medication-basedsafety that includes information on the ‘‘Beer’s criteria for poten-tially inappropriate medication use in older adults” [44], diabetesmellitus focused content, and information on creating, managing,and evaluating CDS programs. The content in the Clinfowikiranges from alerts and reminders to order sets and data displayssuch as flowsheets. Fig. 1 shows a sample rule from theClinfowiki.

3.1.1. GovernanceThe Clinfowiki is hosted by The Informatics Review.1 After cre-

ating a unique user ID and password, anyone can read, edit, create,or even delete content from the site. A wiki system administratorhas additional system-level privileges that allow him to ‘‘block”users who do not use the site appropriately, roll-back the contentto a previous version, or even to ‘‘lock” a page so that no one elsecan edit the content. Content on the site is ‘‘approved” by the com-munity of interested people who have registered with the site. Thereis no formal ‘‘voting” procedure, rather there are mechanisms thatallow individuals to ‘‘discuss” particular pages, to modify contentin a way that allows the group to reach consensus, and to createadditional content to further clarify or refine the content.

The Clinfowiki is designed to be a resource for anyone inter-ested in clinical decision support to both learn about and sharetheir knowledge and content. The idea is that by involving as manydisparate people, with as many different opinions as possible, ex-tremely high-quality information and content will emerge.

Fig. 1. A medication safety related clinical decision support rule in the Clinfowiki.

A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346 337

3.1.2. TechnicalThe Clinfowiki utilizes the open-source platform developed by

MediaWiki (http://www.mediawiki.org) for the Wikipedia project.Currently only human readable content is being developed andshared on the site. Therefore, before using the content within anyspecific clinical information system, a user would first have totranslate the content into their proprietary format, map all theclinical concepts to their local vocabulary or ‘‘orderable catalog”,and test the resulting intervention thoroughly.

In addition to the features and functions mentioned above, theMediaWiki software allows users to: upload files to the site, to beautomatically notified whenever a specific page of contentchanges, to view all previous versions of a page, to see who madespecific changes to a page and when those changes were made.

3.1.3. ActivityAs of 1 September 2008 there were 547 ‘‘articles” relating in

some way to clinical informatics and 577 registered users of thesite. To date, most of the content pages have been created by a verysmall number of users and there have not been a large number ofedits or modifications to most pages. For example, there have onlybeen 2.85 edits per page on average. However, for every edit, therehave been 456 views, suggesting that consumption of the Clin-fowiki far outpaces production. That said, because of the open nat-ure of the Wiki, we do not have any reliable mechanisms for

determining how frequently, if at all, content on the Clinfowiki isactually adopted and used within clinical information systemsand, while feedback has been positive, we are not aware of anysites which have adopted the content and put it into productionin the clinical information systems within their organization.

3.2. Case study 2: Partners HealthCare eRooms

3.2.1. GovernancePartners HealthCare System is an integrated care delivery sys-

tem in Massachusetts founded by two major teaching hospitals,Massachusetts General Hospital and the Brigham and Women’sHospital. Partners also includes several community hospitals and3500 outpatient practitioners. At Partners, governance of clinicalstandards and clinical decision support happens at several levels,ranging from system-wide quality improvement initiatives (whichPartners refers to as ‘‘High Performance Medicine”) to site andpractice specific governance. Each of these governance entitiesmake decisions and produce knowledge which affects clinical deci-sion support at Partners, which presents substantial knowledgemanagement challenges.

To help address these and other challenges, Partners establisheda knowledge management team in 2003 which develops tools,infrastructure, and processes to enable successful management ofknowledge for clinical decision support. The team initially defined

338 A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346

a set of processes and functional requirements to support all stagesof the knowledge management life-cycle including governance,transparency, collaboration, decision capture, workflow manage-ment, life-cycle management, editing, and measurement of processand outcomes [45]. Subsequently, the team established a docu-ment library of all the specifications of content in production thatcould be harvested from shared file areas, and clinical informationsystem source code, to achieve the necessary transparency to man-age the vast array of content already integrated in clinical systems.This internally developed document library, the Partners Knowl-edge Portal, enables inventory, search, and retrieval of all knowl-edge specification documents by several taxonomies, withlimiting by filters and keyword searching. In parallel, the teamworked with clinical leadership to develop new governance struc-tures, subject matter expert panels to develop new content, andknowledge engineering resources to participate in the prioritiza-tion, design, and decision making process. Fig. 2 shows the searchinterface for the portal, which is available to anyone at Partners.This helped ensure that knowledge assets were updated in a timelymanner and that these assets were aligned with the quality, safety,disease management and efficiency business drivers for the healthsystem.

3.2.2. TechnicalBased on these early internal development efforts, it quickly

became apparent that a knowledge management infrastructuremore sophisticated than shared file areas was needed. After a re-view of the marketplace for such tools, the team implementedEMC/Documentum’s eRoom and Content Management Services(EMC Corporation, Hopkinton, MA). The Content Management Ser-vices enable life-cycle management, metadata tagging, publishing,

Fig. 2. Search interface for the partner

versioning, and auditing of content. eRoom supports a variety ofprocesses including project management, virtual-asynchronouscollaboration, voting, decision capture, document versioning,scheduling and overall team coordination. These features wereconsidered essential to support efficient decision making, decisioncapture, and decision auditability among all stakeholders engagedin the KM life-cycle. These stakeholders include clinical leadership,subject matter expert panels, knowledge engineers, developers,analysts and project management personnel.

Together, these features provide a Web 2.0 environment wherean interdisciplinary community of subject matter experts andknowledge engineers can collaboratively review, discuss and engi-neer decision support content.

3.2.3. ActivityThere are now some 40 eRooms in production and over 350

contributing members. These collaboration spaces serve a rangeof teams including site specific teams (i.e. order set management),enterprise KM teams (i.e. patient safety reporting), application KMteams (i.e. ambulatory electronic medical record content), popula-tion KM teams (i.e. geriatric, pediatric), and diagnosis-specificteams (i.e. Diabetes, Cardiac., Obstetrics).

eRooms at Partners support a transparent decision making pro-cess for knowledge-base development in key areas of clinical careand medication management. Historically, it has been difficult toorganize face-to-meetings or determine the previous rationale forknowledge-base design without days spent searching MicrosoftOutlook folders and fileshares. Every collaboration space is man-aged by a designated steward, typically a knowledge engineer,who tees up the questions and proposed drafts for expert panel re-view. Each eRoom supports a range of multidisciplinary experts

s knowledge management portal.

A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346 339

who are authorized by the Clinical Content Committee. In addition,program management, issue tracking and scheduling of ongoingproactive updates are managed in these eRooms.

Discussions range in duration from hours to months dependingon complexity, and may yield as many as fifty different postings asdesign decisions are resolved by acclamation or votes are taken. Adiscussion on a prevention screening reminder, such as colorectalscreening, might address risk stratification rules, effective interpre-tation of the guidelines, appropriate content of messages in clinicalinformation systems, and so forth. Fig. 3 shows a vigorous onlinediscussion and how the steward draws decisions to a close. Theknowledge engineer is responsible for guiding expert panels effi-

Fig. 3. Guided discussion of a colon cancer screening guideline in the partners know

ciently toward consensus. The Partners Knowledge ManagementTeam focuses on clinical knowledge that reflects commonly ac-cepted practices that are important to remember but easy to forgetin the care process. This strategy facilitates rapid consensus devel-opment among expert clinicians. More importantly, the PartnersKnowledge Management team has determined that these expertpanels cannot efficiently participate in the creation and updateprocess without an accountable design steward who is responsiblefor maintaining his/her knowledge domain.

Each night, a report of the day’s activity in the eRoom is sent toall participants. Each entry is identified with a hyper-link so thatthe room members can efficiently navigate to the key points in

ledge management portal eRooms (clinician panel names obscured for privacy).

340 A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346

any discussion thread to participate. After a set of decisions havebeen made in a particular clinical domain, the knowledge engineerformalizes these decisions in a knowledge specification. This spec-ification is then passed to the responsible knowledge engineeringparty to edit and encode into production utilizing the appropriatetool. After that, a document is generated, tagged, and published tothe portal to ensure that the library is up to date as a byproduct ofthe knowledge update or creation process. Over the 3 years sincethe infrastructure was implemented, the KM team has witnessedmaturation in the processes and effective tool use. Templates havebeen developed to streamline the design and kick-off of new col-laboration rooms. Participation by clinician experts remains high,even though it is largely voluntary.

3.3. Case study 3: Epic Community Library

Epic Systems Corporation maintains a website for the use oftheir customers called the Epic UserWeb. The UserWeb was builtto facilitate communication both between Epic and their custom-ers as well as between individual customers. In addition to trainingmaterials, software documentation, event schedules and discus-sion forums, the UserWeb has a Community Library. The Commu-nity Library provides a forum for customers to share clinicalcontent for the Epic suite of healthcare software products. Thismethod of exchange has provided new customers with a wealthof premade materials to use as a starting point for building theirlocal systems as well as offering established customers an avenue

Fig. 4. Epic Community Lib

to share newly developed content. Epic publishes to the Library aswell, including reports, clinical decision rules (called Best PracticeAlerts) and order sets (known as SmartSets).

3.3.1. TechnicalThe UserWeb is hosted by Epic at their corporate offices in

Verona, Wisconsin using Microsoft Sharepoint Server 2007. Cus-tomers choose which content can be shared and arrange with theirEpic technical services contacts to have the materials posted. Asmall team of administrators at Epic then post the content. Thisteam is also responsible for the day-to-day operations of the site.Any customer that has agreed to share content can see and down-load materials from the Library.

Users can search for pieces of content by type, version, contrib-uting organization or by text word. Fig. 4 shows the search inter-face. Clicking on a result takes the user to a detail page showingthe specifics of how to create that material in the user’s system.Depending on the type of content, users can import it directly fromthe Library (in the case of certain documentation templates), im-port using downloaded spreadsheets (for SmartSets), or see thespecific parameters needed (for Best Practice Alerts or other clini-cal reminder tools). In the final case, the site provides an easymethod for printing the material. Fig. 5 shows an example of thedetail page for a documentation template for wrist pain. Userscan import this template directly into their clinical system.Although some of this content is provided by Epic, most of it iscontributed by users of Epic’s software, thus allowing other Epic

rary search interface.

Fig. 5. Sample documentation template in the Epic Community Library.

A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346 341

customers to benefit from each others content development ef-forts. Epic is experimenting with commenting and discussion ofthe materials by piloting an online feedback mechanism on 50SmartSets. If this functionality is popular, it would be possible toopen it up to other pieces of content in the Library. To date, therehas been only limited activity with this functionality.

3.3.2. ActivityThere are approximately 450,000 pieces of content in the Epic

Community Library. These range from flowsheet rows (for docu-menting discrete physiologic data), to documentation tools andtemplates, to SmartSets for both outpatients and inpatients, toclinical decision support rules such as Best Practice Alerts. All ofthe content is human readable, with about 160,000 pieces of thedocumentation tools allowing direct importing from the Libraryinto the users’ systems. Table 1 shows a breakdown of the contentin the Community Library by type. The UserWeb currently has

Table 1Type and count of decision support artifacts in the EPic Community Lirary.

Content Type Artifacts

Clinical decision support alerts 15,000Flowsheet documentation components 16,000Order items 18,000Workflow tools 13,000SmartSets (inpatient and outpatient) 42,000Clinical report tools 45,000Documentation templates 170,000Analytical reports 500

12,000 active users, with the vast majority of them having accessto all of the content in the Library.

4. Discussion

The three case studies presented here: the Clinfowiki, PartnersHealthCare’s eRooms and Epic’s Community Library have in com-mon an overlapping purpose of enabling sharing of clinical deci-sion support content, but the efforts differ in significant ways.The most critical differences among the three sites are their in-tended audiences and contributors. The Clinfowiki is open to thepublic, while Partners eRooms and Epic’s Community Library arerestricted to Partners staff and Epic customers, respectively. In partas a result of their differing audiences, the sites also differ in theformat of content they contain. Since the Clinfowiki can make noassumptions about the clinical information systems of its users,content on the site is free-form and human readable. Since all usersof the Epic Community Library, by contrast, necessarily user Epic’sclinical information system, content posted there is in an Epic spe-cific form which can be directly imported into Epic’s products.

Table 2 summarizes the similarities and differences betweenthe three systems studied. Although the systems differ in theiraudience, content and technical platform, they share several com-mon principles grounded in Web 2.0 philosophy. First, in each case,every user who can access the system can also contribute new con-tent and suggest or perform revisions to existing content. This is incontrast to the central development paradigm where only a smallnumber of authorized contributors develop content which is thenconsumed by a large number of users. Second, all three systemsalso include a mechanism for users to comment on or review con-

Table 2Comparison of the three case study systems.

Clinfowiki eRoom Epic CommunityLibrary

Host Informaticsreview

Partners HealthCare Epic systemscorporation

Access Anyone Partners staff Epic customersTechnology

PlatformMediaWiki Documentum eRoom Microsoft sharepoint

Content format Rich text Spreadsheets,documents, XML

Epic specific

Direct import No No YesContent types Alerts and

remindersAlerts and reminders,order sets, order items,drug dosing tools,documentationtemplates, carepathways, referenceinformation, reports

Alerts and reminders,flowsheets, orderitems, workflow tools,order sets, reports,documentationtemplates

Users 480 350 12,000

342 A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346

tent and make these reviews publicly available. Third, each systemoffers full-text search as well as metadata and tagging to allowusers to find relevant content.

Each of the three systems profiled here has had some measureof success. However, sharing clinical decision support content ingeneral, and particularly by means of Web 2.0 technologies andconcepts, poses some significant issues and challenges. These is-sues are addressed in the subsequent sections and, for purposesof discussion, are grouped into five categories: issues for contribu-tors of CDS content, issues for users of this content, issues for hostsof this content, technical issues of format and issues relating tometadata.

4.1. Issues for contributors of CDS content

There are important issues for both contributors and users ofcontent, many of which can represent important barriers to shar-ing content. Perhaps the most important of these relate to gettingcredit for contributions. If contributors do not benefit academi-cally, socially, or financially, for example from contributing, theymay not have a sufficient incentive to actually do so. Furthermore,as such content gets more complex, entering it and keeping it up todate will become ever more complex and time-consuming forthose who must develop the rules. Organizations may also feel thathaving such content in place represents a competitive advantagethat would be lost if made available to competitors. Another prob-lem is that clinical sites with a large number of rules, such asRegenstrief Institute (Indianapolis, IN) and Partners, have foundthat there are many interactions between rules, which adds enor-mous complexity. A further key issue is when to consider contentproprietary. Some have advocated sharing ‘‘basic” sets of decisionsupport, with more complex decision support remaining proprie-tary. This is the model for many types of software, for example.However, the distinction between ‘‘basic” and ‘‘advanced” is neces-sarily blurry. Furthermore, it is unclear what rules should beshared. Many clinical content vendors exist in this space, and forobvious reasons they are likely to resist making content freelyavailable, especially if they perceive it as ‘‘theirs.” But virtuallyall healthcare organizations rely on one or more of these knowl-edge vendors, raising the question of when content should be con-sidered derivative. Another vexing issue is that in many largeorganizations, it is unclear who is or should be responsible for sign-ing off on organizationally approved contributions.

Finally, a key problem for contributors is the legal one. What if arule is used by someone and it causes harm? Are legal disclaimerssufficient to prevent a lawsuit? Can the organization holding the

rules indemnify or hold harmless those who contribute rules, andeven if they do, would this indemnification hold up in court? Asof now there is little precedent to definitively answer thesequestions.

It is important to note that many of these issues are inherent inany effort to share decision support content and are not specific toWeb 2.0. In a 2007 editorial about the risks and benefits of service-oriented architecture (SOA) in clinical information systems, Drs.Nadkarni and Miller point out that ‘‘[clinical decision support] ser-vices are not ‘licensed practitioners,’ so that legally, the patient’shealth care provider (clinician or institution) is responsible foroverriding any erroneous advice provided by an SOA service.”[46] While this argument (which is the prevailing one as it relatesto decision support in general) would limit the legal risk and liabil-ity of those who share clinical decision support content, theauthors further comment that ‘‘[t]he latter circumstances may inand of themselves inhibit reliance on external clinical SOA ser-vices.” While it may be the case that some consumers would bereluctant to rely on external content which is not guaranteed byits providers, we are not fully convinced by this argument. Thesame principles apply to medical textbooks, reference materialsand clinical guidelines, whose authors and publishers make nowarranty or guarantee of accuracy and take no responsibility forharm resulting from factual errors, yet they are widely used inmedicine.

4.2. Issues for users of CDS content

Users face a number of important problems as well. It may behard to tell exactly whether and how the content has been usedand validated, or what its benefits are. Organizations must alsostruggle with how much internal vetting they must do beforeincorporating any particular piece of decision support content intotheir clinical systems. It may be especially hard to predict what theinteractions with other rules may be. Small hospitals in particularwill have resource limitations in making such decisions. Anotherissue is that the content may or may not be designed with suffi-cient specificity to make it easy to use—for example, a rule may re-fer to a drug class like the ACE inhibitors, but may not specify whatdrugs are in that group, and not all organizations will have the con-cept of drug families already mapped in their local drug terminol-ogy. It is unclear as well how the content should be modified andimproved. For example, if a user finds a problem with a rule, whatis their responsibility to report or fix it? If they do wish to fix it,who has editing authority (i.e. is it just the author, the holder ofthe rules, the editor, or the users)? Another complex set of prob-lems occur with dealing with local modifications. It is essentiallycertain that users will want to modify some rules at least to someextent to deal with local issues such as site specific formularies,antibiotic susceptibilities and referral patterns, but tracking all ofthese, as well as automatically updating rules when they are chan-ged, would be extremely complicated, difficult or even undesirable.While the tools exist to do such tracking, it would place a substan-tial burden on the rule holders.

Indeed, it is not clear what proportion of effort for developingclinical decision support is in getting the core clinical facts right(which would be assisted by Web 2.0-mediated decision supportsharing) and what proportion is in dealing with local, site specificimplementation issues. In a 2004 editorial, Drs. Waitman and Mill-er argue that there are a host of local issues involved in implement-ing guidelines, ranging from ‘‘local, clinically expert champion[s]. . .

who will customize the national guideline to be compatible withlocal capabilities and practices and, more importantly, take owner-ship of guideline evolution locally over time” to local consensusabout guideline implementation and mechanisms for advertisingthe guideline and measuring effectiveness [47]. In fact, they esti-

A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346 343

mate that ‘‘90% of the effort required for successful guidelineimplementation is (and must be) local, and the remaining 10% ofthe effort involves ‘getting the document right.’” We by no meansdiscount the importance of local ownership and adaptation ofguidelines and while we are not fully persuaded that it is possibleto quantitatively estimate the breakdown between that effortwhich might be shared and that which must necessarily be per-formed locally, we would argue that even if the part that mightbe shared is small, it certainly exists, and must be completed be-fore the local tailoring efforts can even begin. Further, Web 2.0methods could be used not only to collaboratively engineer theguideline content, but also to carry out local peer-review activitiesand then share local experiences using the content which couldhelp inform and expedite future local efforts at implementingand maintaining the content.

Users have an additional set of legal issues. What responsibilitydo they have if they take a tool, or piece of clinical content, from apublicly available database, and it has some issue that results inpatient or system harm? Will the disclaimers provided give userssufficient assurance so that they can be confident in using the rulesthat have been made available? Where does the liability fall if aprovider within an organization can be shown not to have followeda rule that came from elsewhere, and how much internal vettingdo they have to demonstrate that they have done for such rules?If not addressed satisfactorily, the legal issues alone could provelimiting, and might be sufficient to keep smaller organizationsfrom utilizing such resources.

4.3. Issues for organizers and hosts of CDS content

One of the critical issues in establishing effective processes andprocedures for web-based knowledge development, sharing, anduse, is to provide rules of the road in several key areas. As the casestudies above illuminate, it is important to have defined author-ship, review, and contribution policies, policies for revising, andtailoring knowledge, policies for resolving conflicts of opinion,effective methods to use feedback to update content and create alearning knowledge system, and appropriate and sufficient legalprotections in place to allow knowledge sharing to occur. In thissection, we review each one of these issues in turn.

4.3.1. Contributing knowledge and contentAt one extreme, the web allows anyone to contribute content

for all to see. In the case of knowledge being contributed to supportclinical decision making in the process of care delivery by licensedprofessionals, there is a requirement that the knowledge is pro-vided in a way that allows an end-user to determine its credibilityand validity with full transparency regarding the source of the con-tent (author(s), citations, or potentially even primary data sources),posting date, anticipated update date, and any feedback providedby current providers using the content in their clinical practice.As a matter of practicality, content should only be contributed bylicensed professionals in the discipline for which the knowledgeis intended to be used (e.g. licensed physicians providing knowl-edge for physician clinical decision support). A second level ofqualification might be envisioned where only board-certified, orsimilarly qualified professionals, are allowed to make contribu-tions. To promote full transparency of medical knowledge usedin clinical decision support, we suggest that any provider, or anypatient, should have full access to web-based repositories to in-spect the knowledge and content contained therein.

4.3.2. Implementing shared knowledgeOnce clinical knowledge content is posted in a web environ-

ment for sharing, methods must be in place to determine how tobest use that knowledge in the context of an end-user’s local prac-

tice, and what distinguishes allowed modifications or ‘tailoring’ ofthe knowledge for local implementation from that which is essen-tial or core to the knowledge itself. In a fashion analogous to the‘‘Z” segments of an HL7 message, an end-user must know how totake a knowledge object from the shared repository, and interpretit or tailor it for his or her own use. Local practice patterns maytypically modify the workflow of certain knowledge objects, butthey should not modify the core aspects of the knowledge objectitself.

4.3.3. Updating shared knowledgeUndoubtedly, we will experience a progression of methods by

which shared knowledge will be updated and refined. Initially,the tailoring of knowledge for local implementation as describedabove may be based on subjective differences of opinion or localpatterns of practice. However, over time, more complex feedbackmechanisms may be developed. One might envision an automatedmechanism where local experience with content can be sharedwith its developer in real-time. For example, a developer mightshare a package of rules for preventive care and screening in dia-betic patients. As the rules are implemented in local sites, overriderates and reasons might be communicated back to this developerin addition to being analyzed and reviewed locally. From the aggre-gate data, the developer might notice that the rule for performing(or referring for) foot exams is overridden more frequently than ex-pected and, further, that the most common reason for override is‘‘patient has bilateral leg/foot amputation”. Based on this feedback,the developer could revise the rule to exclude patients with re-corded lower limb amputation and, through an automated or man-ual process, alert users of these rules to the newly modifiedversions.

4.3.4. Creating a learning knowledge systemThe processes described above for tailoring knowledge for local

implementation, for providing subjective and objective feedbackon knowledge usage and impact, and the evolution of the evi-dence-base from primary research itself, can all be used to drivea process for learning and continually refining the knowledge inshared knowledge repositories. Feedback on the utility of knowl-edge used in clinical decision support must first be provided tothe end-user so that she may assess the value directly of the rele-vant knowledge to her practice, for example in a clinical decisionsupport quality dashboard. Secondarily, feedback on clinical utilityand impact on clinical processes and outcomes must be directedback to the authorized knowledge stewards so that they may usethis feedback, along with updates from primary research, to revisethe content in shared knowledge repositories. In this manner, wecan not only improve the ‘efferent’ limb of the knowledge life-cy-cle, but also dramatically improve the ‘afferent’ limb of feedbackto improve the knowledge-base.

4.3.5. Legal issues with knowledge sharing and re-useIn current practice, knowledge is typically shared and re-used

freely and willingly among healthcare providers. Increasingly, ascare is provided more often by care teams, potentially distributedgeographically, it will be critical for all participants to have accessto the best knowledge, and a clear care plan, which may be in-formed by clinical decision support. So long as clinical decisionsupport systems are not subject to medical device certification pro-cedures the policies above should suffice. Whenever knowledge isused for clinical decision making in a ‘closed-loop’ setting, how-ever, where a licensed professional does not have the opportunityto review, and potentially intercept untoward, recommendationsfor clinical decision support we suggest that the knowledge in thatsetting will require validation and attestation from an authorizedbody to assert its validity. To promote knowledge sharing and re-

344 A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346

use as has been the tradition in healthcare, we suggest that thosesharing knowledge and best practices, which are adopted and usedin usual and customary practice, be indemnified from any liabilityassociated with use of such knowledge.

4.3.6. Enhancing existing systems with Web 2.0 featuresIt is worth nothing that developing Web 2.0 systems for sharing

decision support content need not necessarily be a de novo exer-cise. Indeed, many of the non-Web 2.0 systems described in thebackground section of this paper could, with relative ease, haveWeb 2.0 capabilities added. For example, the Columbia Arden Syn-tax MLM repository is currently a static list of MLMs that were inuse at Columbia as of March, 1997. This list could be expandedto include the ability for users to comment on or rate the MLMs,as well as to contribute their own MLMs to the repository, thusbecoming a Web 2.0 sharing system.

4.4. Technical issues: format

A critical technical issue for Web 2.0 sharing of clinical decisionsupport is the ideal format to use. A spectrum of approaches exists:

1. Free-form narrative text. Most clinical guidelines are writtenthis way, and AHRQ’s repository of guidelines (http://www.guidelines.gov) contains many of them. Although this isa natural and easy to understand approach for representingknowledge, guidelines written this way often contain ambigui-ties, and this approach is distant from implement ability

2. Disambiguated human readable text. This represents a moreformal, but still free-text approach to representing guidelines.Clinical algorithms and flow charts are often written at thislevel

3. Unambiguous, machine interpretable guidelines. A number ofknowledge representation formalisms, such as the previouslymentioned Arden Syntax and GLIF have been developed toencode knowledge in a machine interpretable form

4. Fully implemented, executable code. Many guidelines are hard-coded in programming languages, and it’s often necessary totranslate Level 3 guidelines to this level in order to implementthem

There is some degree of cross-over between these levels. Forexample, the best guidelines read as though they were written atLevel 1, but actually are sufficiently unambiguous and detailed toqualify as Level 2. Further, clinical systems may support the inter-pretation and execution of guidelines written in Level 3 (only ahandful of commercial clinical systems have this capability today,usually implemented as an Arden Syntax compiler).

It’s unclear exactly how knowledge should be represented in aWeb 2.0 environment. There are two competing goals that affectthis decision: the desire for the knowledge to be easily read anddiscussed (which suggests Levels 1 or 2) and the desire for theknowledge to be easily implemented in clinical systems with aminimum of further translation (which suggests Level 3 or 4).The best compromise is probably to work at both Levels 2 and3, with clinical subject matter experts collaborating at Level 2and knowledge engineers translating to and collaborating at Level3. Level 1 seems undesirable because these discussions alreadyoccur during the guideline development process and because Le-vel 1 is so distant from actionable, implementable decision sup-port. Level 4 is unattractive because content at this level isoften specific to a particular installation of a particular clinicalsystem, significantly limiting the pool of possible collaborators.Research in automatic and facilitated techniques for translatingbetween levels is ongoing, and may someday render this decisionmoot [48–50].

4.5. Technical Issues: Metadata

As knowledge libraries grow, organizing and searching thembecomes more difficult. A key tool for both organization and searchis metadata—data which describes the content in the system. Somecore metadata likely to be attached to content in most Web 2.0 CDSsystems might include:

� Bibliographic information, such as the authors of the content,when it was developed, when it was updated and the sourceof the recommendations (such as a clinical guideline or resultsof a published trial)

� The clinical purpose of the guideline, such as the disease or con-dition it is focused on, the objective behind developing the CDScontent, the expected users and care settings in which the con-tent is likely to be used, the target population for the guideline,inclusion and exclusion criteria and the expected health out-come from deploying the content

� How the content was developed, including methods for collect-ing, curating and qualifying the evidence it is based on

� The format of the content (such as alert, reminder, clinical ruleor order set), and the type of clinical information system thecontent is likely to be used in

� How the content was validated and tested� Information for implementers of the content, including antici-

pated barriers and enablers, strategies and performancemeasures

Several standard metadata models for decision support arti-facts have been proposed, either on their own or as part of broad-er knowledge representation formalisms such as DeGel [51], EON[52], HGML [53], GUIDE [54], GLIF and HELEN [55]. One particu-larly comprehensive and influential model is the Guideline Ele-ments Model (GEM) [56], which has been adopted by otherapproaches including GUIDE and GLIF [57]. A key issue regardingmetadata is how structured it should be. The most basic type ofmetadata are keywords applied to decision support content—thisis known in Web 2.0 parlance as tagging, and is popularly used innon-clinical systems such as Del.icio.us or Flickr. It is sometimesadvantageous to impose further structure on metadata. For exam-ple, keywords can be constrained to come from controlled termi-nologies and vocabularies and tags can be extended to attributevalues pairs. As an example of these two structural elements,one might require that all content be tagged with an attribute va-lue pair about the related disease state (e.g. ‘‘disease state = diabe-tes mellitus”), and might further require that the value beencoded in a specified terminology (i.e. diabetes mellitus couldbe represented with SNOMED CT code 73211009). When theseterminologies have internal semantic networks, this providesadditional benefits. For example, SNOMED specifies a diabetesmellitus concept, but it also specifies an endocrine disorder con-cept, and encodes the ‘‘is-a” relationship that diabetes is an endo-crine disorder. So a properly developed metadata system whichused SNOMED CT could automatically return diabetes contentwhen a user searched for endocrine disorders without requiringthat the content be explicitly tagged as dealing with an endocrinedisorder.

Once a metadata model has been established, responsibility forapplying appropriate metadata must be assigned. Clearly contribu-tors of content are important stakeholders in this process, but itmay also make sense for the community to work together by,say, adding additional tags to content posted by others. Dependingon how the content is organized and hosted, there may also beroom for a librarian role—certainly the process of assigning meta-data to decision support content is analogous to a librarian’s cata-loging role.

A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346 345

5. Summary and conclusions

We believe the case for Web 2.0 as a tool for collaborating onclinical decision support content appears strong, particularly forcollaborative content development within an organization. How-ever, as discussed, there are a number of organizational, legal, pol-icy and technical issues which remain unresolved. The use of Web2.0 for collaborative development and sharing of clinical decisionsupport is relatively recent, and it seems that clarity on many ofthese issues may be achieved simply by gaining more experience.That said, some of the issues require real work. For example, a clearliability framework, which may be developed through a combina-tion of legislation and regulation, along with the development of abody of case law, is needed to clarify legal issues. A better under-standing of the potential commercial or competitive value of deci-sion support content is needed to inform discussions onintellectual property issues. And further technical work is neededto solidify knowledge representation formats and metadata. Thesechallenges notwithstanding, the future for Web 2.0 in clinical deci-sion support seems bright, and we are eager to see how such tech-nologies, and their use, evolve.

Acknowledgments

This work was supported in part by AHRQ CERT Grant1U18HS016970, AHRQ Research Grant R01-HS015169, AHRQ con-tract HHHSA29020080010 and NLM Research Grant R56-LM006942-07A1. The funding agencies were not involved in theresearch or preparation of the manuscript. We are grateful toVenkatesh Janakiraman of Epic Systems for his help in developingthe usage statistics presented for the Epic Community Library aswell as to Aiman Alarawabeh who created much of the drug–druginteraction content on the ClinfoWiki as his final project for a grad-uate course in Medical Informatics.

References

[1] Balas EA, Weingarten S, Garb CT, Blumenthal D, Boren SA, Brown GD.Improving preventive care by prompting physicians. Arch Intern Med2000;160(3):301–8.

[2] Barnett GO, Winickoff R, Dorsey JL, Morgan MM, Lurie RS. Quality assurancethrough automated monitoring and concurrent feedback using a computer-based medical information system. Med Care 1978;16(11):962–70.

[3] Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducingthe frequency of errors in medicine using information technology. J Am MedInform Assoc 2001;8(4):299–308.

[4] Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma’Luf N, et al. The impact ofcomputerized physician order entry on medication error prevention. J Am MedInform Assoc 1999;6(4):313–21.

[5] Burt CW, Hing E. Use of computerized clinical support systems in medicalsettings: United States, 2001–2003. Adv Data 2005;2(353):1–8.

[6] Classen DC. Clinical decision support systems to improve clinical practice andquality of care. JAMA 1998;280(15):1360–1.

[7] Doolan DF, Bates DW, James BC. The use of computers for clinical care: a caseseries of advanced US sites. J Am Med Inform Assoc 2003;10(1):94–107.

[8] Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J,et al. Effects of computerized clinical decision support systems on practitionerperformance and patient outcomes: a systematic review. JAMA2005;293(10):1223–38.

[9] Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinicaldecision support systems on physician performance and patient outcomes: asystematic review. JAMA 1998;280(15):1339–46.

[10] Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-basedclinical decision support systems on clinician performance and patientoutcome. A critical appraisal of research. Ann Intern Med 1994;120(2):135–42.

[11] Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practiceusing clinical decision support systems: a systematic review of trials toidentify features critical to success. BMJ 2005;330(7494):765.

[12] Kuperman GJ, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G, et al.Medication-related clinical decision support in computerized provider orderentry systems: a review. J Am Med Inform Assoc 2007;14(1):29–40.

[13] McDonald CJ. Protocol-based computer reminders, the quality of care and thenon-perfectability of man. N Engl J Med 1976;295(24):1351–5.

[14] Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A Roadmapfor National Action on Clinical Decision Support. J Am Med Inform Assoc2007;14(2):141–5.

[15] Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, et al. Grandchallenges in clinical decision support. J Biomed Inform 2008;41(2):387–92.

[16] Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: asystematic review of rigorous evaluations. Lancet 1993;342(8883):1317–22.

[17] Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematicreview: impact of health information technology on quality, efficiency, andcosts of medical care. Ann Intern Med 2006;144(10):742–52.

[18] Johnson D, Pan E, Walker J, Bates DW, Middleton B. Patient Safety in thePhysician’s Office: Assessing the Value of Ambulatory CPOE. Calif HealthcareFound 2004.

[19] Shortliffe EH, Patel VL, Cimino JJ, Octo Barnett G, Greenes RA. A study ofcollaboration among medical informatics research laboratories. Artif IntellMed 1998;12(2):97–123.

[20] Hripcsak G. Arden syntax for medical logic modules. MD Comput1991;8(2):76–8.

[21] Health Level 7. Arden Syntax for Medical Logic Systems Standard Version 2.6.Ann Arbor MI: Health Level 7; 2007.

[22] Peleg M, Tu S, Bury J, Ciccarese P, Fox J, Greenes RA, et al. Comparingcomputer-interpretable guideline models: a case-study approach. J Am MedInform Assoc 2003;10(1):52–68.

[23] Patel VL, Allen VG, Arocha JF, Shortliffe EH. Representing clinical guidelines inGLIF: individual and collaborative expertise. J Am Med Inform Assoc1998;5(5):467–83.

[24] Ohno-Machado L, Gennari JH, Murphy SN, Jain NL, Tu SW, Oliver DE, et al. Theguideline interchange format: a model for representing guidelines. J Am MedInform Assoc 1998;5(4):357–72.

[25] Peleg M, Boxwala AA, Ogunyemi O, Zeng Q, Tu S, Lacson R, et al. GLIF3: theevolution of a guideline representation format. In: Proc AMIA Symp 2000. p.645–9.

[26] Wang D, Peleg M, Tu SW, Boxwala AA, Ogunyemi O, Zeng Q, et al. Design andimplementation of the GLIF3 guideline execution engine. J Biomed Inform2004;37(5):305–18.

[27] Ram P, Berg D, Tu S, Mansfield G, Ye Q, Abarbanel R, et al. Executing clinicalpractice guidelines using the SAGE execution engine. Medinfo 2004;11(Pt1):251–5.

[28] Scalise D. Physician order entry. IMKI to success. Institute for medicalknowledge implementation. Hosp Health Netw 2002;76(10):24–6.

[29] OpenClinical. Methods and tools for the development of computer-interpretable guidelines: Arden Syntax; 2002 Available from: http://www.openclinical.org/gmm_ardensyntax.html [cited 2008 March 07].

[30] Jenders RA. The Arden syntax for medical logic systems: frequently askedquestions; 2006 Available from: http://cslxinfmtcs.csmc.edu/hl7/arden/faq.html. [cited 2008 March 07].

[31] Jenders RA, Sujansky W, Broverman CA, Chadwick M. Towards improvedknowledge sharing: assessment of the HL7 Reference Information Model tosupport medical logic module queries. In: Proc AMIA Annu Fall Symp 1997. p.308-12.

[32] O’Reilly T. What is Web 2.0: design patterns and business models for the nextgeneration of software; 2005 Available from: http://www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html. [cited 2008 January 18].

[33] Gray A, Haahr M. Personalised, collaborative spam filtering. proceedings of thefirst conference on email and anti-spam (CEAS). 2004.

[34] Giles J. Internet encyclopaedias go head to head. Nature 2005;438(7070):900–1.

[35] Laningham S. Developer works interviews: Tim Berners-Lee originator of theWeb and director of the world wide Web consortium talks about where we’vecome, and about the challenges and opportunities ahead; 2006 Available from:http://www-128.ibm.com/developerworks/podcast/dwi/cm-int082206.txt.[cited 2008 March 07].

[36] Wikipedia contributors. Web 2.0. wikipedia, the free encyclopedia; 2008Available from: http://en.wikipedia.org/w/index.php?title=Web_2.0&oldid=196983179. [cited 2008 March 09].

[37] Wikipedia contributors. History of wikis. Wikipedia, the free encyclopedia;2008 Available from: http://en.wikipedia.org/w/index.php?title=History_of_wikis&oldid=191676342. [cited 2008 March 07].

[38] Giustini D. How Web 2.0 is changing medicine. BMJ 2006;333(7582):1283–4.[39] Kamel Boulos MN, Wheeler S. The emerging Web 2.0 social software: an

enabling suite of sociable technologies in health and health care education.Health Info Libr J 2007;24(1):2–23.

[40] McLean R, Richards BH, Wardman JI. The effect of Web 2.0 on the future ofmedical practice and education: darwikinian evolution or folksonomicrevolution? Med J Aust 2007;187(3):174–7.

[41] Jenders RA, Hripcsak G, Sideli RV, DuMouchel W, Zhang H, Cimino JJ, et al.Medical decision support: experience with implementing the Arden Syntax atthe Columbia-Presbyterian Medical Center. In: Proc Annu Symp Comput ApplMed Care 1995. p. 169-173.

[42] Johnson KR, Freeman SR, Dellavalle RP. Wikis: the application of Web 2.0.. ArchDermatol 2007;143(8):1065–6.

[43] Middleton B, Anderson J, Fletcher J, Masarie FE Jr., Leavitt MK. Use of theWWW for distributed knowledge engineering for an EMR: the KnowledgeBankconcept. In: Proc AMIA Symp 1998. P. 126-30.

[44] Fick DM, Cooper JW, Wade WE, Waller JL, Maclean JR, Beers MH. Updating theBeers criteria for potentially inappropriate medication use in older adults:

346 A. Wright et al. / Journal of Biomedical Informatics 42 (2009) 334–346

results of a US consensus panel of experts. Arch Intern Med2003;163(22):2716–24.

[45] Glaser J, Hongsermeier T. Managing the investment in clinical decisionsupport. In: Greenes RA, editor. Clinical decision support: the roadahead. Burlington, MA: Elsevier, Inc.; 2006.

[46] Nadkarni PM, Miller RA. Service-oriented architecture in medical software:promises and perils. J Am Med Inform Assoc 2007;14(2):244–6.

[47] Waitman LR, Miller RA. Pragmatics of implementing guidelines on the frontlines. J Am Med Inform Assoc 2004;11(5):436–8.

[48] Wang D, Peleg M, Tu SW, et al. Representation primitives, process models andpatient data in computer-interpretable clinical practice guidelines: a literaturereview of guideline representation models. Int J Med Inform 2002;68(1-3):59–70.

[49] Georg G, Colombet I, Jaulent MC. Structuring clinical guidelines through therecognition of deontic operators. Connecting medical informatics and bio-informatics: proceedings of MIE2005, the XIXth international congress of theEuropean federation for medical informatics; 2005.

[50] Georg G, Séroussi B, Bouaud J. Extending the GEM model to support knowledgeextraction from textual guidelines. Int J Medical Info 2005;74(2-4):79–87.

[51] Shahar Y, Young O, Shalom E, et al. A framework for a distributed, hybrid,multiple-ontology clinical-guideline library, and automated guideline-supporttools. J Biomed Inform 2004;37(5):325–44.

[52] Tu SW, Musen MA. Modeling data and knowledge in the EON guidelinearchitecture. Medinfo 2001;10(Pt 1):280–4.

[53] Hagerty CG, Pickens D, Kulikowski C, Sonnenberg F. HGML: a hypertextguideline markup language. In: Proc AMIA Symp 2000. p. 325–9.

[54] Ciccarese P, Caffi E, Boiocchi L, Quaglini S, Stefanelli M. A guidelinemanagement system. Medinfo 2004;11(Pt 1):28–32.

[55] Haschler I, Skonetzki S, Gausepohl HJ, Linderkamp O, Wetter T. Evolution ofthe HELEN representation for managing clinical practice guidelines.Available from: <http://www.klinikum.uni-heiidelberg.de/fileadmin/inst_med_biometrie/med_Informatik/helen3/evolution_helen_methods.pdf>.[cited 2008 August 15].

[56] Shiffman RN, Karras BT, Agrawal A, Chen R, Marenco L, Nath S. GEM A proposalfor a more comprehensive guideline document model using XML. J Am MedInform Assoc Am Med Inform Assoc 2000:488–98.

[57] Peleg M, Tu S, Bury J, et al. Comparing computer-interpretable guidelinemodels: a case-study approach. J Am Med Inform Assoc 2003;10(1):52–68.