Rating the Ratings

Part 4 – Rating HIS Raters

By Vince Ciotti & Elise AmesHIS Professionals, LLC

(with a lot of help from Mr. HIStalk,and many of you readers)

Criteria for RatingsThis week we’re going to rate the 3 best-known HIS reviewers (KLAS, Black Book & Peer60) based on our key criteria listed below:

1. Number – how many responses were received: 5, 50, 500?2. Job Class – who responded: CIOs, RNs, MDs, ancillaries?3. Date – when were responses received: in 2015? Since 2010??4. Size – what size were the hospitals: CAH, community, AMCs…5. Products – which of vendors’ various products were rated?

And the toughest questions to ask, let alone answer:6. Cost – how much do they charge for various reports?7. Revenue – how much do vendors have to pay to participate?8. Ratio – any relationship between vendor revenue & ratings?9. Special Fees – can vendors pay more to improve ratings?

We’ll try to rate them just as objectively as they rate HIS vendors...

1. Number of Responses• First and most important, how many responses were received? A

simple question one would think, but the answers are complex:

KLAS - uses a system of check marks next to their scores to indicate “Konfidence Levels:”

= 30 organizations

= 20 organizations

= 15 organizations -- = low KonfidenceNB: “Konfidence levels as used by KLAS do not imply the strict meaning found in statistical textbooks.”

Black Book – each reports lists how many responses it is based on. E.g.: for their large hospital complimentary 2014 EHR survey (we couldn’t afford the 2015 one!), the “number of responses validated” for AMCs & large hospitals over 250 beds in size was 1,985, a very big #!

Peer60 – Mr. HIStalk used their RaaS (Research as a Service) tools to evaluate Epic recently (http://histalk.com/machform/view.php?id=33121

), and received 202 responses from Epic clients.

Confidence Levels• How many responses should

earn one’s “confidence?”• We’re sure statisticians

could do a 20-page episode with the answer, like this page from the Engineering Statistics Handbook:

• Our bottom line: the more the merrier! Black Book’s 1,985 replies are far more significant than KLAS’ 30+…

• And look for a high % of the target: Epic has about ≈800 clients, so Peer60’s 202 is 25%, not a bad sampling.

(Sample) Size Matters…Besides the absolute # of responses, it is equally important to know how many different hospitals the responses were received from, so raters should publish both figures. Do any of our raters give both?• A CIO described a detailed KLAS report she received on her EHR

vendor (with about 250 clients), which she felt was a good report. But the report did not reveal the total number of responses or hospitals. At the back were 208 interesting comments from CIOs, CMOs, CNIOs, etc., however there is no way to estimate the number of responses or hospitals from the # of comments…

• Same with Black Book – if every 5 or 10 of their 1,985 responses worked at the same hospital – then this impressive number would only reflect only 400 or 200 sites. Since there are ≈800 hospitals over 300 beds, it would represent 50% or 25%...

• Lastly, Peer60 also doesn’t say how many separate facilities were in their 202 responses – could be 100, could be 50, could be…

2. Job Titles• The job title of respondents tells you a lot about their rating of an

HIS: think CEO vs CFO vs CIO vs MD vs RN vs biller vs registrars…

KLAS – the high-level “Best In Class” report makes no mention of job titles in its 90 pages. The CIO’s report on their vendor does list the job title for each of the 208 comments at the bottom: Manager, CIO, Nurse, CMIO, VP/Other Exec, Analyst, etc. But no totals are given…

Black Book – the 30-page 2014 “Large Hospital EHR Survey” which they provide for free does not even mention a single job title. Zero, nada, nothing, zilch...

Peer60 – Mr. HIStalk’s “epic” Epic report starts with the job titles of all 202 survey responders:• “22 chief executive officers• 13 chief financial officers• 96 chief information officers• 39 chief medical officers• 32 chief nursing officers”

It then graphs which title influenced buying Epic the most, how CMOs rated Epic less than CNOs & CEOs, and are less likely to buy it again…

Do Job Titles Matter?What does it matter what the respondents’ job titles are? We had a fascinating chat regarding this subject with Jeremy Bikman, who worked at KLAS for 7 years before leaving to form Peer60. The net:• Who are the primary users of an EHR? Easily, it’s physicians and

nurses who are stuck using it for their entire work day. Their EHR ratings are far more valuable than a CEO, CFO or even CIO’s…

• Whose opinion would you value the highest when rating LIS systems? Why, Lab Directors of course. And PACS systems? Radiologists! The same for Access, RCM, ERP, etc. It’s the end users in various depts. who use it the most that know the most.

• This is not to denigrate CIOs’ ratings, since they probably have the best overall feeling for a vendors’ performance. But their expertise is far more technical (data center, releases, interfaces, etc.) than end users’ (features, user-friendliness, # of clicks, etc).

So it’s best to see ratings by job titles: of these 3, only Peer60 did.

3. Date of Responses• The time-frame of responses can be very significant in terms of

releases, updates, and vendor M & As like Cerner & Evident’s.

KLAS – the “Best In Class 2015/2016” date seems self-evident, but no specific date range is given. Their detailed report on our CIO’s vendor listed the exact month & year (e.g.: June 2016) for all the 208 comments at the bottom, which is ideal! The oldest was from December 2015…

Black Book – the 2014 Large Hospital EHR Survey they provide for free on their web site states on its cover:• “Survey Period: Q3

2013 – Q1 2014.” They include tables of

the # of responses by year from 2010 to 2014, as well as “Basic vs Fully Functional” EHR scores by year for HIS vs MDs.

Peer60 – Mr. HIStalk’s EPIC report came out early this year, so it’s very current. But no dates for responses are given so future readers may not know its original time frame…

4. Size of Facility• From <25-bed Critical Access (CAH) to 1K+ bed AMCs & multi-IDNs,

the size of respondents can have a large impact on ratings.KLAS – only one table in the 90-page “Best In Class” report gives specific bed sizes: on page 19 is a table that rates “Community HIS (1-50 beds).” An odd range considering CAH is defined as under 25 beds... The detailed report on our CIO’s vendor made no mention of bed sizes.

Black Book – their 2014 “Large Hospital EHR Survey” states on its cover: “Hospitals over 300 Beds.” But nothing more granular like:• Community Hospitals

(300+ beds) – which are less challenging than:

• AMCs (300+beds), much more complex than community, let alone:

• Multi-hospital IDNs, some with thousands of beds.

Peer60 – only rated Epic, that only sells to 300+ bed hospitals, so bed size was not specified. However, with Epic Connect, there are now a growing # of small & CAH users…

5. Products/Release• When Consumer Reports rates Honda cars, do they combine the

$10K 1980 Civic along with the $200K 2017 NSX? We hope not!

KLAS – Odd product names in Best in KLAS:• “Meditech C/S” = #4 -

the old Client/Server, not the new Rel. 6?

• For community EHRs, #1 is Cerner’s “Mill. Power-Chart/Com-munity Works” – which one? Big diff!

• Community HIS #1 is “Meditech C/S (6.x)” – is it C/S or Rel 6.x??

Black Book – confusing product names in their ranking of EMRs too:• #2 = “Cerner Millennium

PowerChart/Community Works” – which one of the two do they mean?

• #3 = “Meditech Magic” is rated better than:

• #4 “Meditech 6.x,” but no mention made of C/S?

These products are very different in terms of age, platform, processing…

Peer60 – they’re lucky since there is only one Epic (EpicCare), and Judy doesn’t buy products. In the future Epic Connect and the new remote hosting might be worth segregating…

6. FeesA difficult question to ask, and even harder to answer, is what these

rating agencies charge vendors & providers for their products.

KLAS Black Book Peer60Fees for vendors to participate

Yes – but $s not listed on web site…

No $0Fees to providers for current reports

No – but provider must participate

Yes - last year’s are complimentary $0

Fees to vendors & consult. for reports

Yes – but $s not listed on web site…

Yes - $995 per report for 2016 $0

Ratio between vendor fees & rank?

Vendor revenue unknown…

None (no vendor fees) N/A

Special fees for “coaching”

Yes, per oral reports from vendors

No, per oral reports from vendors No

7. Report CardsHere’s our “confidence level” for the 3 HIS rating agencies

reviewed – using KLAS’ check-mark scoring approach:

KLAS Black Book

Peer60

Number of responses – total and as % of the target market

Job title of respondents – with scores from each job class

Date of responses – year range, date of each response

Size of hospitals – by bed size, with scores from each range

Products – separate scores for vendor’s various products

Money – fees for reports, participation, special fees…

BOTTOM LINE – our totally subjective ranking (which is probably exactly how they would rate our report!):

Next Week• We’d greatly appreciate any feedback on how we rated these 3

leading HIS reviewers before we cover the dozens of MD raters…• If we survive that feedback, next week will give the results of Mr.

HIStalk’s survey, with over 60 responses received already. It lets you score many of the same questions on these raters at:

• http://histalk.com/machform/view.php?id=33836• Then a detailed episodes will cover physician practice reviews:

– FPM, SK&A, Software Advice, MD Buyline, AmericanEHR, etc.• Lastly, suggestions on how to “do it yourself” – which may be far

cheaper (only your users’ time) to get very relevant info on:– Your size facilities, your state, your product, your apps…

• For compliments, please contact Elise Ames at:– eames@hispros.com, 413.329.6925

• For complaints, please contact Vince Ciotti at:– vciotti@hispros.com, 505.466.4958