ashish k. jha, md, mph · ashish jha, md, mph ashish k. jha, md, mph director, harvard global...
TRANSCRIPT
-
1
August 21, 2018
Georg Roeggla
Associate Editor, The BMJ
Dear Dr. Roeggla,
Thank you for giving us an opportunity to revise our manuscript “Patient Outcomes and Accreditation in
U.S. Hospitals: An Observational Study.” We appreciate the issues raised by the reviewer and the editor’s
requests, and we are grateful for the opportunity to respond. Additionally, since the original submission of
our manuscript, we have obtained 2015 Medicare data and updated our results. Below are the comments
we received from the editor and reviewers (in italics) and our responses (in bold).
We believe that the reviewer comments were very helpful and addressing them has strengthened the
manuscript. We hope you agree. If there are any further issues or concerns that we can address, please do
not hesitate to contact us.
Sincerely,
Ashish Jha, MD, MPH
Ashish K. Jha, MD, MPH
Director, Harvard Global Health Institute
K.T. Li Professor of Global Health
Professor of Medicine
Harvard Medical School
-
2
14-Jun-2018
Dear Dr. Jha
Manuscript ID BMJ.2018.044844 entitled "Patient Outcomes and Accreditation in U.S. Hospitals: An
Observational Study"
Thank you for sending us your paper. We sent it for external peer review and discussed it at our
manuscript committee meeting. We recognize its potential importance and relevance to general medical
readers, but I am afraid that we have not yet been able to reach a final decision on it because several
important aspects of the work still need clarifying.
We hope very much that you will be willing and able to revise your paper as explained below in the
report from the manuscript meeting, so that we will be in a better position to understand your study and
decide whether the BMJ is the right journal for it. We are looking forward to reading the revised version
and, we hope, reaching a decision.
Please remember that the author list and order were finalized upon initial submission, and reviewers and
editors judged the paper in light of this information, particularly regarding any competing interests. If
authors are later added to a paper this process is subverted. In that case, we reserve the right to rescind
any previous decision or return the paper to the review process. Please also remember that we reserve
the right to require formation of an authorship group when there are a large number of authors.
Yours sincerely,
Georg Roeggla
**Report from The BMJ’s manuscript committee meeting**
These comments are an attempt to summarise the discussions at the manuscript meeting. They are not
an exact transcript.
Manuscript meeting 14.06.2018
Elizabeth Loder (chair), Angela Wade (stats), Wim Weber, John Fletcher, Georg Roggla, Tiago Villanueva.
Decision: Ask for revision
Thank you for the opportunity to address the questions and comments raised by the committee and
the reviewers.
-
3
The committee was interested in the topic of your research. The following concerns were mentioned.
• We think your paper is of interest not only to US readers. But we think it would be very helpful to
explain The Joint Commissions work in more detail. What is the process? What happens during
accreditation?
Thank you for this suggestion. We have now added additional information and references about the
Joint Commission as well as more details in general on the process of accreditation by other
accrediting organizations (AOs) and the state survey to the introduction. Specifically, we write the
following:
“The U.S. hospital accreditation process varies between state survey agencies and AOs.16–20
A
hospital that elects to undergo survey by a state agency can expect an annual, unannounced,
onsite inspection that determines their accreditation status. These reviews vary in length and
usually ensure that the hospital has people and policies needed to provide adequate quality
care. AOs are required to inspect hospitals at least every three years. TJC, for example,
performs unannounced onsite surveys for its clients every 18 to 36 months, whereas DNV GL,
a newer AO, performs annual onsite inspections. Additionally, AOs tend to provide more
structure, consulting with hospitals on how to prepare for an inspection and often have
additional quality metrics that they choose to examine. During the on-site inspection,
surveyors observe a broad range of hospital operations, but the focus is still largely on
structural factors and processes.19
There is little focus by AOs on whether the hospital is
achieving good outcomes.”
• Where are the joint commission accreditations performed apart from the US?
The international branch of the Joint Commission currently accredits over 1000 organizations in over
60 countries. The Joint Commission accredits many aspects of the health care system, from hospitals
and ambulatory surgical centers to nursing homes. We have added this information to the
introduction.
“Moreover, the international branch of TJC currently accredits over 1000 organizations in over
60 countries outside the U.S.7”
• Please explain the differences in the process when accreditation is performed by other
organisations.
Thank you – in response to the committee’s prior suggestion regarding explaining the Joint
Commission and the accreditation process, we added information regarding other AOs as well to the
introduction. Please see response above.
• The committee shares the concern that the information on accreditation status and mortality
and readmission were not collected within the same time period. Accreditation status is collected
between 2014 and 2017 and outcome data between January 1, 2014 and November 30, 2014. Please
explain how hospitals that undertake surveys in 2016 or 2017 are likely to perform better on patient
outcomes in 2014?
Thank you for this important comment, which has been echoed below by one of the reviewers.
-
4
We initially performed our mortality and readmissions analyses with the most contemporary
Medicare data we had available at the initiation of our project. We included all hospitals accredited
between 2014-2017 to analyze all U.S. hospitals. Hospitals have typically undergone accreditation for
years, if not decades, and therefore, we assumed that a single survey visit would not necessarily lead
to improved patient outcomes. As the Joint Commission states on their website, “accreditation is
about continuous quality — not ‘passing’ a survey”,1 and we expected that patients receiving care at
hospitals with accreditation would receive a standard quality of care irrespective of the most recent
inspection date. In fact, that’s consistent with the evidence that while there are transient changes in
hospital practice around the inspection date, the outcomes in a hospital does not vary based on when
the last inspection occurred.
It was with this empirical background that we focused on all hospitals that were accredited using the
most recent data we have and compared it to outcomes using the most recent data we had. We were
further assured of this approach given that it is very rare for hospitals to change accrediting
organizations.
Since our initial analyses, we have obtained 2015 Medicare data. We performed a sensitivity analysis
using this more recent data and our results are quite similar overall (Table 1, Table 2). There is no
significant difference in medical and surgical mortality for privately accredited hospitals versus those
undergoing a State Survey. There is no difference in surgical readmission rates and a trend toward
lower medical readmissions for privately-accredited hospitals, which is not significant based on our
pre-specified p-value. We have included the results of this sensitivity analysis in our manuscript
(Appendix Tables 6 and 7).
Finally, we performed one additional sensitivity analysis using the updated 2015 Medicare data on a
subset of hospitals, specifically those that were accredited in 2014 only (Table 3, Table 4). This most
directly addresses the issue of whether accreditation is associated with better outcomes in the
subsequent period.
This approached substantially decreased our hospital sample and the number of hospitalizations.
These results are largely similar with similar effect estimates and no statistically significant differences
based on our pre-specified p-values. There is a significantly higher surgical mortality rate for non-TJC
hospitals compared to TJC hospitals. Here, because we are working with a small sample size, it
appears that non-TJC privately accredited hospitals that happened to be accredited in 2014 had
particularly poor surgical outcomes that one year – a trend we don’t see when we examine
subsequent years of non-TJC accredited hospitals. We have included the results of this sensitivity
analysis in our manuscript (Appendix Tables 8 and 9).
Table 1. 30-Day Risk-Adjusteda Mortality and 30-Day Risk-Adjusted
a Readmission Rates for Hospitals
Accredited by an Accrediting Organization (AO) vs. Reviewed by a State Survey Agency (2015 Medicare
data)
AO
% [95% CI] State Survey
% [95% CI] Difference
% [95% CI] p-value
b
30-Day Mortality
Medical 10.3%
[10.2% - 10.4%] 10.6%
[10.3% - 10.9%] -0.3%
[-0.7% - 0.04%] 0.08
-
5
Surgical 2.14%
[2.09% - 2.18%] 2.07%
[1.8% - 2.4%] 0.07%
[-0.2% - 0.4%] 0.67
30-Day Readmissions
Medical 22.3%
[22.2% - 22.4%] 22.9%
[22.4% - 23.4%] -0.6%
[-1.1% - -0.1%] 0.02
Surgical 14.9%
[14.7% - 15.2%] 14.3%
[13.0% - 15.6%] 0.7%
[-0.6% - 2.0%] 0.30
aAdjusted for patient and hospital characteristics and HRR fixed effects. bA p-value less than 0.0125 was considered significant.
Table 2. 30-Day Risk-Adjusteda Mortality and 30-Day Risk-Adjusted
a Readmission Rates for Hospitals
Accredited by TJC vs. non-TJC (2015 Medicare data)
TJC
% [95% CI] Non-TJC
% [95% CI] Difference
% [95% CI] p-value
b
30-Day Mortality
Medical 10.3%
[10.2% - 10.4%] 10.2%
[10.0% - 10.5%] 0.04%
[-0.2% - 0.3%] 0.75
Surgical 2.15%
[2.10% - 2.20%] 2.2%
[2.0% - 2.3%] -0.03%
[-0.2% - 0.1%] 0.72
30-Day Readmissions
Medical 22.3%
[22.1% - 22.4%] 22.4%
[22.0% - 22.8%] -0.1%
[-0.5% - 0.3%] 0.51
Surgical 15.0%
[14.7% - 15.3%] 15.1%
[14.3% - 15.9%] -0.1%
[-0.9% - 0.7%] 0.82
aAdjusted for patient and hospital characteristics and HRR fixed effects. bA p-value less than 0.0125 was considered significant.
Table 3. 30-Day Risk-Adjusteda Mortality and 30-Day Risk-Adjusted
a Readmission Rates for Hospitals
Accredited by an Accrediting Organization (AO) vs. Reviewed by a State Survey Agency (2015 Medicare
data for 2014 accreditation)
AO
% [95% CI] State Survey
% [95% CI] Difference
% [95% CI] p-value
b
30-Day Mortality
Medical 10.3%
[10.17-10.45%] 10.8%
[10.40-11.20%]
-0.49%
[-0.96- -0.02%] 0.04
Surgical 2.0%
[1.90-2.07%] 2.1%
[1.70-2.47%]
-0.10%
[-0.54-0.34%] 0.66
30-Day Readmissions
Medical 21.9%
[21.67-22.16%] 22.8%
[22.13-23.43%]
-0.87%
[-1.61- -0.13%] 0.02
Surgical 13.7%
[13.24-14.13%] 13.1%
[11.38-14.77%]
0.61%
[-1.27-2.49%] 0.52
aAdjusted for patient and hospital characteristics and HRR fixed effects. bA p-value less than 0.0125 was considered significant.
Table 4. 30-Day Risk-Adjusteda Mortality and 30-Day Risk-Adjusted
a Readmission Rates for Hospitals
Accredited by TJC vs. non-TJC (2015 Medicare data for 2014 accreditation)
TJC Non-TJC Difference p-value
b
-
6
% [95% CI] % [95% CI] % [95% CI]
30-Day Mortality
Medical 10.3%
[10.15-10.39%] 11.1%
[10.40-11.74%]
-0.80%
[-1.50- -0.10%] 0.03
Surgical 2.1%
[1.99-2.12%] 2.8%
[2.36-3.14%]
-0.70%
[-1.11- -0.28%] 0.001
30-Day Readmissions
Medical 21.9%
[21.66-22.11%] 22.8%
[21.50-24.17%]
-0.96%
[-2.33-0.42%] 0.17
Surgical 13.9%
[13.48-14.28%] 15.0%
[13.16-16.86%]
-1.13%
[-3.08-0.81%] 0.25
aAdjusted for patient and hospital characteristics and HRR fixed effects. bA p-value less than 0.0125 was considered significant.
• Please discuss pros and cons of the accreditation process. Is marketing a major issue?
Due to the increased focus on healthcare quality and patient safety, there has been greater
expectation from hospitals and hospital leadership on providing evidence of their commitment. This
in turn has led to the use of hospital accreditation as a way to market high quality healthcare.
While this has been part of the U.S. system for many decades, use of hospital accreditation is now
starting to take off globally. The assumption is that if hospitals pay large sums of money to
accreditors, they will get the “gold seal of approval” which can assure consumers quality and lead to
substantial revenues for hospitals. Our findings call into question to what degree this gold seal of
approval really means better outcomes.
There is no doubt accreditation has value. It forces hospitals to examine their staffing, their
processes, and to pay some attention to quality. There is some evidence that accreditation leads to a
bit higher rates of adherence to guidelines that are being tracked.
However, the cons are that the process is expensive, cumbersome, and some clinicians feel it takes
away from their focus on clinical care. And of course, it is a major source of marketing by the
hospitals and an excellent business model for the accrediting agencies – which also sell consulting
services to help hospitals get ready for their inspectors and consulting services which help hospitals
address problems found by the inspectors.
• We liked that you look at patient outcomes and not just processes of care.
Thank you for your comment. We agree that it is important to look at all aspects of care, especially
patient outcomes.
• Could you give some more information, perhaps in a box or elsewhere, about the differences
between hospitals that are and are not JC accredited, e.g. size, geographic location, patient mix, etc.
-
7
Thank you for this comment. We agree that there are baseline differences in hospitals that are
accredited. Table 1 highlights some of the differences in patient characteristics (age, race, sex,
Medicaid eligibility, and comorbidities) and hospital characteristics (size, ownership, teaching status,
geographic location, urban location, critical access hospital, and ICU). We have added additional
information to the results section to further describe these and a new table in the appendix.
“Compared to hospitals accredited by TJC or non-TJC organizations, hospitals reviewed by
State Survey were more often smaller, non-teaching institutions, more likely to be located in
rural areas and lacking an intensive care unit (ICU). TJC hospitals are more likely to be larger,
teaching institutions, located in urban locations, and in the northeast and south as compared
to hospitals accredited by non-TJC organizations or reviewed by a State Survey (Appendix
Table 3). Across the three groups, the majority of patients were white and female, and about
a quarter were dual eligible with similar comorbidity profiles.”
• It sounds like the accreditation visit is intended to improve the standard of care and is an
intervention in its own right. It would be important therefore to compare hospitals on a like for like time
period either before an accreditation visit or a suitable time after an accreditation visit.
Thank you for this comment.
Dr. Jena and colleagues looked specifically at whether hospitals undergoing the TJC survey had better
outcomes during the week of the survey. They found significantly lower mortality rates for patient
admitted to hospitals during TJC survey weeks compared to those admitted during non-survey weeks
(three weeks before and three weeks after the survey week). Therefore, while there was lower
mortality during the weeks of the survey, the mortality rates returned to baseline after the survey
was complete.
The primary goal of our study is to understand not the short-term effects of inspectors arriving at the
hospital – but whether, at a national level – different approaches to accreditation (private, expensive
accreditation that is quite exhaustive versus a lighter private accreditation versus a state survey) are
associated with differential patient outcomes.
As mentioned above, we performed an additional sensitivity analysis using the updated 2015
Medicare data on a subset of hospitals, specifically those that were deemed in 2014 only (Table 3,
Table 4). Our sample size decreases significantly by including only a subset of hospitals. The results are
qualitatively similar although we do find in this one subset of hospitals that surgical mortality is lower
for those at TJC vs non-TJC. We have included this in our revised manuscript (Appendix Table 8 and 9).
• The committee thought your conclusions may be too strong for the data presented.
Thank you – we have modified our conclusions to reflect the limitations of our data.
• Causal inference should not be made from observational data.
Thank you – we have modified the wording of our discussion.
-
8
• Confidence intervals for the differences between AO and State Survey and between TJC and non-
TJC in tables 2-5 should be given. The confidence limits should be considered in the interpretation of the
results.
Thank you for this suggestion. Table 2-5 have been updated to include the differences with their
confidence intervals.
First, please revise your paper to respond to all of the comments by the reviewers. Their reports are
available at the end of this letter, below. Please also respond to the additional comments by the
committee.
In your response please provide, point by point, your replies to the comments made by the reviewers and
the editors, explaining how you have dealt with them in the paper.
Thank you for the opportunity to respond to comments and suggestions by the committee and
reviewers to improve our manuscript.
-
9
** Comments from the external peer reviewers**
Reviewer: 1
Recommendation:
Comments:
Please note that I am at the same university as the study authors and have collaborated with study
authors Jha and Orav on previous studies. However, I have expertise in analysis of Medicare claims data,
have linked TJC accreditation data to Medicare data in a previous study (other than the study I am
currently reviewing and my own study that is referenced, I am unaware of prior studies that perform this
linkage), and (through my prior work) am familiar with the broader literature on the role of hospital
accreditation in influencing patient outcomes.
This article studies the relationship between hospital accreditation and patient outcomes in U.S.
hospitals. The hospital accreditation business is large and is an important mechanism through which
hospitals attempt to ensure the provision of high quality health care. The study links data on Medicare
beneficiaries who are hospitalized for a set of general medical or surgical conditions to data on
accreditation by either the Joint Commission (TJC), other non-TJC accrediting organizations, or state
surveyors. The basic finding of the article is that accreditation bears very little relationship with 3 main
patient outcomes: 30 day mortality, 30 day readmissions, and patient experience with hospital care.
Process measures of quality were not assessed.
I am unaware of other studies which comprehensively analyze the relationship between hospital
accreditation, in particular accreditation by TJC, and patient outcomes, at least using data as recent as
that in this article. At the time of my own work on this issue - which studied how patient outcomes
change during the exact dates of TJC hospital inspections - my review of the literature was consistent
with how these study authors have interpreted current studies. Findings on the role of accreditation in
hospitals are mixed, usually focus on process measures of outcomes, and do not study the relationship
with patient outcomes. I would add that most if not all previous studies are smaller in size than the
current study. The question that's asked is important given the size of this industry, the importance that
is placed by regulators on hospital accreditation, the efforts that hospitals go through to gain
accreditation, and the belief that accreditation is necessary, despite many other factors that may be as
or more important to hospitals in incentivizing them to provide high quality care.
Thank you for your review and the opportunity to respond to your comments to improve our
manuscript.
I have several comments / suggestions:
1. The BMJ audience is international. While the existing Discussion briefly highlights work on the
role of hospital inspection in other countries, many readers of the BMJ will be unfamiliar with how
hospital accreditation works in the US and what is the role and importance of TJC. US reader will know
this information but other ex-US readers may not. I would include more detail on these issues to make
them salient to international readers.
Thank you for this comment. The editorial committee had similar suggestions and we have now added
additional information and references to the introduction.
-
10
2. I generally found the empirical approach to be appropriate. An alternative approach with
longitudinal data could be to look at how hospital outcomes change before and after accreditation. Aside
from issues with data availability, this analysis may be difficult to conduct if most hospitals stay
accredited and rarely is it the case that hospitals switch in and out of accreditation status.
Thank you for the comment. We agree with the reviewer that a longitudinal analysis with time-
dependent covariates would have been ideal – but as the reviewer points out, few hospitals change
their exposure (type of accreditation) and when they do, it is usually for other reasons that are likely
correlated with outcomes. Therefore, such a strategy would not give us reliable answers in this
instance.
3. The authors can consider adding more discussion as to why accreditation is not associated with
better outcomes. There are at least a few reasons that come to mind. US hospitals compete with one
another within local or regional markets. A primary thesis of competitive markets is that competition
drives improvements in quality. While there are asymmetries of information in health care, one reason
accreditation may not be associated with patient outcomes is that hospitals are already competing on
presumably patient outcomes and other factors. In addition to this explanation, the malpractice system
in the US may already exert influence on hospitals to provide high quality care. Finally, insurers have
information about quality of care in hospitals and may use their bargaining power to influence decisions
of hospitals to invest in quality of care. Insurers can exclude low quality providers from their network and
this competitive pressure may induce investments into quality. Finally, it may be the case that
accreditation can improve process measures, but these measures are poorly correlated with the
outcomes the authors study.
Thank you for these comments. We have now added additional information to our discussion to
address these important points.
“It is possible that accreditation by an independent AO is not associated with better patient
outcomes because the focus of AOs has been on improving structural factors and clinical
processes but less on actually improving patient outcomes. Prior work has shown that efforts
at improving clinical processes of care can lead to better patient outcomes,52,53
but this does
not always hold true.54,55
Furthermore, general hospital accreditation has demonstrated mixed
results on patient outcomes.1,29,31,56
Additionally, we did not observe better patient experience
among patients receiving care at an accredited hospital, and in fact, satisfaction was slightly
worse. This is consistent with findings from Sack and colleagues who showed that
accreditation is not associated with better quality as perceived by patients across 73 hospitals
in Germany.57
Again, this may be a result of accreditation by AOs emphasizing measures that
do not directly translate to better patient experience. There are several other explanations for
why accreditation is not associated with better outcomes. Since U.S. hospitals compete within
local or regional markets, competition may be a driver of overall improvement in quality. It is
also possible that the U.S. malpractice system may be exerting influence on hospitals to
provide high quality care. Finally, it is possible that insurance companies have information
about quality of care in hospitals and may use their bargaining power to influence decisions of
hospitals to invest in quality of care. They might exclude low quality providers from their
network, providing additional pressure for hospitals investments into quality.”
-
11
4. Can the authors provide some data on how many hospitals attempt to get TJC accreditation but
do not? If this information is available, they might consider comparing those hospitals that achieved TJC
accreditation versus those that attempted to but did not.
This is an interesting question and the other reviewer asked a similar question. It turns out that a very
small proportion of hospitals are denied accreditation – based on the Joint Commission website, it
appears that in 2015, only 8 hospitals were denied accreditation and another 14 received
accreditation with follow-up. It appears that almost all hospitals are accredited.
To respond to the reviewer’s suggestion, we ran new analyses that compared hospitals that received
accreditation by TJC, those that were initially denied but accredited with follow-up and those that
were denied altogether. We see little difference across the groups though to some extent, the denied
group appears to have the lowest mortality rates (though this is likely an artifact of small numbers).
There are no significant differences in mortality and readmissions between these three groups.
Table 5. Risk-Adjusteda Overall and Star Rating Scores for TJC-Hospitals Accredited vs. Accredited with
follow up vs. Denial of Accreditation (2015 Medicare data)
Accredited
[95% CI]
Accredited with
Follow up
[95% CI]
Denial
[95% CI] p-value
b
30-Day Mortality
Medical 10.3%
[10.22-10.36%] 11.2%
[10.11-12.35%] 9.5%
[8.58-10.33%] 0.09
Surgical 2.2%
[2.12-2.21%] 1.8%
[1.03-2.49%] 1.6%
[0.60-2.65%] 0.38
30-Day Readmissions
Medical 22.3%
[22.18-22.41%] 23.6%
[22.44-24.65%] 22.8%
[21.17-24.45%] 0.13
Surgical 15.2%
[14.91-15.43%] 15.3%
[12.54-18.13%] 15.0%
[10.57-19.38%] 0.99
aAdjusted for patient and hospital characteristics and HRR fixed effects. bA p-value less than 0.0125 was considered significant.
5. Some of the language in the discussion is causal or approaches causal statements. Given that the
study design isn't quasi-experimental, it would be worth while in any revision for the authors to pay close
attention to emphasizing the association of accreditation with patient outcomes, rather than a causal
effect.
Thank you for this suggestion. We completely agree with the reviewer and have now modified our
language to emphasize the association of accreditation with patient outcomes rather than imply a
causal effect.
-
12
Reviewer: 2
Dear authors,
Thanks for an interesting study on accreditation. The study is really appreciated in the effort to illuminate
the effectiveness of accreditation on patient outcomes relevant to many readers around the world.
However, this review have raised a number of concerns that the authors profitably could address to
strengthen the presented study.
Thank you for your helpful comments and for the opportunity to address the comments.
Introduction:
The introduction could benefit from a detailed summary of the existing literature review on the results of
studies on accreditation and patients’ outcome including a description of methodological challenged in
accreditation research performed so far. Furthermore, the authors’ could profitably expand their focus to
research on other accreditation program with similar content and survey methodology than the TJC and
non-TJC.
The authors’ state that no contemporary accreditation data have shown better outcomes and refer to a
study from 2002. However, other more recent studies do not support this argument:
Azagury D, and Morton JM2. “Bariatric Surgery Outcomes in US Accredited vs Non-Accredited
Centers: A Systematic Review.” J Am Coll Surg. 2016 Sep;223(3):469-77.
Gratwohl A,et al, Use of the quality management system "JACIE" and outcome after
hematopoietic stem cell transplantation. Haematologica. 2014 May;99(5):908-15.
Thank you for your comment. We agree that there are several different types of accreditation with
some focusing more narrowly on certain conditions and others that are broader, focusing on entire
hospitals such as The Joint Commission. As the reviewer points out, there have been more recent
papers on accrediting organizations that do not accredit hospitals but rather specific aspects of the
hospital. While our overarching goal was to focus on hospital-level accreditation, we have updated
our introduction to be more general and include these references about department-specific or
condition-specific accreditation. To be more comprehensive with our literature search, we have
included additional references as well.
“There are many existing types of healthcare accreditation that are condition- or specialty-
specific to hospital- and organization-level efforts. The current literature on accreditation
reveals a mixed story of whether accreditation improves processes of care and outcomes2,3,12–
21,4–11. For hospital accreditation specifically, much of the evidence to date has focused
narrowly on the impact of accreditation on structural factors or processes of care, many of
which are emphasized and assessed by TJC.4,6–11,14,22,23
”
Study question:
There is a discrepancy between the phrasing of the third question for patient experience and the
presented results. Thus, two analyses are performed with first merging the accredited hospitals into one
exposure group (AO’s) compared to state surveys and secondary the TJC and non-TJC are assessed
separately.
Thank you for pointing this out. We have changed the wording of the third question to reflect the
presented results.
-
13
“And finally, how does patient experience differ between hospitals accredited by an AO
versus those undergoing a State Survey and between hospitals accredited by TJC versus non-
TJC organizations?”
Methods:
A paper on accreditation requires to incorporate a detailed description of the intervention under
investigation in order for the readers to understand the potential mechanisms on how and why
accreditation may affect patient outcome. This includes among others details on content of the
standards and survey methodology for both the TJC and the non-TJC’s frameworks. Furthermore, an
accreditation programme have several aims besides improving patient outcomes, thus a clarification on
what part of the programs that are most likely to affect patient outcomes is needed.
Thank you for this comment. The editorial committee and the other reviewer also commented on the
need for further information on the accreditation process. Please see above for details but we have
now included additional information and references to help the broad swath of BMJ readers, some of
whom may be less familiar with the accreditation process.
We agree with the reviewer that a hospital accreditation program may have aims beyond assessing
and improving patient outcomes, although surely assuring a high level of quality is near the top of
that list. The accreditation programs such as the Joint Commission primarily measure structural and
process measures during their survey visits as a proxy for high quality health care being delivered at
their hospital. We have attempted to clarify that while structural factors and processes are primarily
measured, the data is mixed on whether even those are improved with accreditation, and that for our
study, we wanted to focus on patient outcomes and experience.
It is an important area of concern that the information on accreditation status and mortality and
readmission were not collected within the same time period. Accreditation status is collected between
2014 and 2017 and outcome data between January 1, 2014 and November 30, 2014. The authors’ need
to explain how hospitals that undertake surveys in 2016 or 2017 are likely to perform better on patient
outcomes in 2014?
Thank you for this important comment, which has been echoed above by the editorial committee.
Please refer to our full discussion above. In brief, we thought that our approach was appropriate
because most hospitals are accredited for long periods of time and prior studies suggest that there
aren’t major changes in outcomes after an accreditation visit. In fact, the Joint Commission and
others argue that the date of accreditation is meant simply as a test in what is largely a broader, long-
term quality assurance program. And, since most hospitals retain their accreditation after each visit, a
hospital accredited in 2016 was almost surely also accredited in 2013 and therefore, the 2016 date is
simply a proxy for this being a Joint Commission accredited hospital over the longer run.
This all said, we performed two additional sensitivity analyses to address the reviewer’s concerns.
First, we have now obtained 2015 Medicare data and have repeated our analyses. We find that results
are similar to the original paper submission (Tables 1, 2 above). Second, we then narrowed our
hospitals to those deemed in 2014 to look at 2015 outcomes (Tables 3, 4 above). In that subset of
hospitals, there was no difference in mortality and readmissions for privately accredited hospitals
compared to those undergoing a state survey. There was significantly lower surgical mortality for
hospitals accredited by TJC vs non-TJC entities.
-
14
To my understanding, the identification of patients and data on mortality and readmission are derived
from Medicare inpatients. Because the study is relevant to readers worldwide, please include a
description of the characteristics of patients eligible for Medicare and possible a reference for further
description. In addition, it is not clear, whether outcomes are restricted to in-hospital data or
independent of place of death and if readmission are to the hospital for the index admission or any
hospital.
Thank you for your comment. We have incorporated the following information into the manuscript
with additional references:
“In the U.S., Medicare is available for people age 65 or older, younger people with disabilities,
and people with end-stage renal disease.30,31
Patients with Medicare often have multiple
chronic conditions and lower median income than the rest of the population.32
”
Patients are selected based on inpatient admission into a hospital, but 30-day mortality is not
conditional on place of death. 30-day readmissions are calculated based on readmission to any
hospital after the index admission. We have added additional text into the manuscript to clarify this:
“Thirty-day mortality is calculated for any death (in-hospital or elsewhere) in the 30 days
following the admission date.”
A description of the collection of data on patient experience should be included in the Methods section.
As clarified in the statistical section, data are collected from April 2015 to March 2016, but it unclear
when data on patients’ experience are gathered. Are “discharge patients” asked on the exact date of
discharge or how is this done in HCAPHS? The paper lacks information on the data collection including
methods for distributing the questionnaire (paper or electronic), whether the procedure were
standardized and therefore similar for all hospitals and finally, by reporting patient response rates
according to exposure groups.
Thank you – we have now included additional information and references about the HCAPHS survey
as well as patient response rates by exposure group.
“The HCAHPS survey is a standardized set of questions given to patients who were discharged
from the hospital who had at least one overnight stay in the hospital as an inpatient, had a
non-psychiatric principal diagnosis at discharge, and were alive at time of discharge. Patients
cannot be surveyed while they are still in the hospital. Sampled patients are surveyed
between 48 hours and 6 weeks after discharge.33
There are four approved modes of
administration, including mail only, telephone only, mail followed by telephone, and active
interactive voice response.34
The HCAHPS survey asks discharged patients 27 questions about
their recent hospital stay, and these questions are grouped and reported in the following 11
publicly reported measures: four composite measures of clinical domain (responsiveness of
hospital staff, pain management, discharge information, and care transition), three
communication measures (with physicians, with nurses, and about medications), two items
related to hospital environment (cleanliness and quietness), and two global measures (overall
hospital rating and likelihood to recommend). The CMS summary star rating scores hospitals
on a one to five-star scale based on the eleven domains in the HCAHPS survey. Response rates
for the three groups were 29% for TJC, 30% for non-TJC, and 34% for State Survey.”
-
15
The specific diagnostic codes included for each medical condition needs to be outlined as the groupings
are not exhaustive e.g. “Metabolic diseases” or “respiratory diseases” the later do obviously not include
COPD. Furthermore, their relevance for 30-day mortality must be presented in more details too.
Thank you for the suggestion. We have updated our Appendix Table 1 in our manuscript to include the
specific diagnostic codes for each condition. We used 30-day mortality as it is a metric collected by
the Centers for Medicare and Medicaid (CMS) and used in numerous studies both in the U.S. and
internationally to evaluate patient outcomes.
Appendix Table 1. Common medical and surgical procedures and their associated diagnosis-related
group (DRG) codes
Condition Code Code Description
Medical Condition
Acute Myocardial
Infarction
280 ACUTE MYOCARDIAL INFARCTION, DISCHARGED ALIVE W
MCC
281 ACUTE MYOCARDIAL INFARCTION, DISCHARGED ALIVE W
CC
247 PERC CARDIOVASC PROC W DRUG-ELUTING STENT W/O
MCC
282 ACUTE MYOCARDIAL INFARCTION, DISCHARGED ALIVE
W/O CC/MCC
246 PERC CARDIOVASC PROC W DRUG-ELUTING STENT W MCC
OR 4+ VESSELS/STENTS
249 PERC CARDIOVASC PROC W NON-DRUG-ELUTING STENT
W/O MCC
283 ACUTE MYOCARDIAL INFARCTION, EXPIRED W MCC
Chronic Heart
Failure
292 HEART FAILURE & SHOCK W CC
291 HEART FAILURE & SHOCK W MCC
293 HEART FAILURE & SHOCK W/O CC/MCC
Pneumonia
194 SIMPLE PNEUMONIA & PLEURISY W CC
193 SIMPLE PNEUMONIA & PLEURISY W MCC
195 SIMPLE PNEUMONIA & PLEURISY W/O CC/MCC
Chronic Obstructive
Pulmonary Disease
190 CHRONIC OBSTRUCTIVE PULMONARY DISEASE W MCC
191 CHRONIC OBSTRUCTIVE PULMONARY DISEASE W CC
192 CHRONIC OBSTRUCTIVE PULMONARY DISEASE W/O
CC/MCC
Stroke
65 INTRACRANIAL HEMORRHAGE OR CEREBRAL INFARCTION
W CC
64 INTRACRANIAL HEMORRHAGE OR CEREBRAL INFARCTION
W MCC
66 INTRACRANIAL HEMORRHAGE OR CEREBRAL INFARCTION
W/O CC/MCC
Sepsis
871 SEPTICEMIA OR SEVERE SEPSIS W/O MV 96+ HOURS W MCC
872 SEPTICEMIA OR SEVERE SEPSIS W/O MV 96+ HOURS W/O
MCC
Esophageal/Gastric
Disease 392
ESOPHAGITIS, GASTROENT & MISC DIGEST DISORDERS W/O
MCC
-
16
391 ESOPHAGITIS, GASTROENT & MISC DIGEST DISORDERS W
MCC
372 MAJOR GASTROINTESTINAL DISORDERS & PERITONEAL
INFECTIONS W CC
371 MAJOR GASTROINTESTINAL DISORDERS & PERITONEAL
INFECTIONS W MCC
Gastrointestinal
Bleeding
378 G.I. HEMORRHAGE W CC
377 G.I. HEMORRHAGE W MCC
379 G.I. HEMORRHAGE W/O CC/MCC
Urinary Tract
Infection
690 KIDNEY & URINARY TRACT INFECTIONS W/O MCC
689 KIDNEY & URINARY TRACT INFECTIONS W MCC
Metabolic Disease 641 NUTRITIONAL & MISC METABOLIC DISORDERS W/O MCC
640 NUTRITIONAL & MISC METABOLIC DISORDERS W MCC
Arrhythmia
310 CARDIAC ARRHYTHMIA & CONDUCTION DISORDERS W/O
CC/MCC
309 CARDIAC ARRHYTHMIA & CONDUCTION DISORDERS W CC
308 CARDIAC ARRHYTHMIA & CONDUCTION DISORDERS W
MCC
243 PERMANENT CARDIAC PACEMAKER IMPLANT W CC
244 PERMANENT CARDIAC PACEMAKER IMPLANT W/O CC/MCC
Chest Pain
313 CHEST PAIN
0
287 CIRCULATORY DISORDERS EXCEPT
Renal Failure 683 RENAL FAILURE W CC
682 RENAL FAILURE W MCC
Respiratory Disease 177 RESPIRATORY INFECTIONS & INFLAMMATIONS W MCC
178 RESPIRATORY INFECTIONS & INFLAMMATIONS W CC
Hip Fracture
481 HIP & FEMUR PROCEDURES EXCEPT MAJOR JOINT W CC
470 MAJOR JOINT REPLACEMENT OR REATTACHMENT OF
LOWER EXTREMITY W/O MCC
482 HIP & FEMUR PROCEDURES EXCEPT MAJOR JOINT W/O
CC/MCC
480 HIP & FEMUR PROCEDURES EXCEPT MAJOR JOINT W MCC
Surgical Procedure
Coronary Artery
Bypass Grafting
236 CORONARY BYPASS W/O CARDIAC CATH W/O MCC
234 CORONARY BYPASS W CARDIAC CATH W/O MCC
233 CORONARY BYPASS W CARDIAC CATH W MCC
235 CORONARY BYPASS W/O CARDIAC CATH W MCC
Open Abdominal
Aortic Aneurysm
Repair
238 MAJOR CARDIOVASC PROCEDURES W/O MCC
237 MAJOR CARDIOVASC PROCEDURES W MCC OR THORACIC
AORTIC ANEURYSM REPAIR
Endovascular
Abdominal Aortic
Aneurysm Repair
238 MAJOR CARDIOVASC PROCEDURES W/O MCC
Colectomy
330 MAJOR SMALL & LARGE BOWEL PROCEDURES W CC
329 MAJOR SMALL & LARGE BOWEL PROCEDURES W MCC
331 MAJOR SMALL & LARGE BOWEL PROCEDURES W/O CC/MCC
Pulmonary
Lobectomy
164 MAJOR CHEST PROCEDURES W CC
165 MAJOR CHEST PROCEDURES W/O CC/MCC
-
17
163 MAJOR CHEST PROCEDURES W MCC
Hip Replacement 470 MAJOR JOINT REPLACEMENT OR REATTACHMENT OF
LOWER EXTREMITY W/O MCC
Finally, the Methods section would be strengthened by a section that clarifies why it is appropriate to
compare hospitals that are accredited with those that are not. In particularly when Table 1 outlines
major differences in size (resulting in huge difference in numbers of hospitalization pr. hospitals; state
survey average 171 vs TCJ = 1253 and Non-TJC=1006), teaching status, urban, Critical assess hospital,
and numbers of intensive care units. The later may have a significant influence on the patient’s chance of
survive.
We agree that there are important baseline differences in hospitals that chose to undergo
accreditation by a private organization such as the Joint Commission (TJC) and others (non-TJC),
versus those hospitals that opt to undergo the complimentary state survey.
Our initial hypothesis was that since hospitals that undergo private accreditation through TJC or non-
TJC organizations are often bigger hospitals with more resources, our results would be biased toward
showing better outcomes at privately accredited hospitals (and at TJC hospitals over non-TJC privately
accredited hospitals). Given that we did not see this difference, we have more confidence in the
finding that there seems to be little advantage of getting a private accreditation or in TJC over other
approaches.
Statistical analysis:
The authors need to clarify the terms dual-eligibility, and CAH status – what is a Critical Assess Hospital
for those not familiar to the US hospital system). Furthermore, it is unclear what population is included
for the outcome of readmission including how competing risk for death is handled in the analyses?
Thank you for this suggestion. We agree the need to clarify several of our terms and have now
incorporated the following into our manuscript:
”Medicaid eligibility was determined using the State Buy-In Coverage Count variable. Any
beneficiary with at least 1 month of state buy-in was considered Medicaid eligible. Dual-
eligibility refers to patients who are eligible for both Medicaid and Medicare. Eligibility for
Medicaid is determined primarily by state-level poverty thresholds. Information on hospital
characteristics was obtained from the American Hospital Association (AHA) annual survey and
Medicare Impact File. Admissions to non–acute care hospitals, federal hospitals, and those
outside the 50 states and the District of Columbia were excluded. Critical Access Hospitals
(CAHs) were included in our study and are defined by statute in the United States. Their key
characteristics are that they are small (have 25 or fewer inpatient beds) and are rural (located
more than 35 miles from another hospital). These rural hospitals face substantial burdens in
providing access to high quality care. To help reduce their financial burden, these hospitals are
reimbursed on a cost basis.45
”
“We followed the standard approach to calculating readmissions as developed by CMS.50
The
numerator includes any unplanned readmissions to a non-federal, short-stay, acute-care or
-
18
critical access hospital within 30 days after discharge. Multiple readmissions are counted
once, and same-day readmissions for the same principal diagnosis at the same hospital are
excluded. The denominator includes beneficiaries 65 years or older who are hospitalized at
non-federal, short-stay, acute care or critical access hospitals. A readmission may also serve as
an index hospitalization. There are numerous exclusion criteria which include: death during
admission, discharge against medical advice, cancer hospitalization, or lacking continuous
enrollment in Medicare for at least 30 days post discharge.”
For calculating readmissions, we followed CMS guidelines, which do currently do not address
competing risk. Our mortality measure uses medical and surgical conditions as a selection mechanism
and to adjust for differences in patient severity but does not assign mortality to any one cause.
Similarly, our readmission measure excludes from the denominator those who died during the
admission.
The study would be strengthened considerably by adding a number of stratified analyses to explore the
results for different exposure groups. This includes e.g. stratification by calendar periods, level of
compliance achieved by the hospitals (accredited, accredited with excellence etc.) and how many cycles
of accreditation the hospitals have completed. Furthermore, it could be interesting to see the result for a
sensitivity analysis restricting to hospitals with ICU.
Thank you for this suggestion. We have performed several additional sensitivity analyses as suggested
by the committee and reviewer. Above, we have performed the following analyses:
- By calendar period (2015 outcomes data for hospitals deemed in 2014; Tables 3, 4 above;
manuscript Appendix Tables 6, 7)
- By level of compliance achieved by the hospitals (only available for TJC-accredited hospitals;
Table 5 above; manuscript Appendix Tables 8, 9)
- By hospitals with an ICU (Tables 6, 7)
Unfortunately, there is no publicly available data regarding how many cycles of accreditation hospitals
have completed.
Table 6. 30-Day Risk-Adjusteda Mortality and 30-Day Risk-Adjusted
a Readmission Rates for Hospitals
with an ICU Accredited by an Accrediting Organization (AO) vs. Reviewed by a State Survey Agency
(2015 Medicare data)
AO
% [95% CI]
State Survey
% [95% CI]
Difference
% [95% CI] p-value
b
30-Day Mortality
Medical 8.6%
[8.55-8.70%]
8.9%
[8.27-9.43%]
-0.23%
[-0.82-0.37%] 0.46
Surgical 1.7%
[1.62-1.72%]
1.6%
[1.19-2.00%]
0.07%
[-0.34-0.48%] 0.72
30-Day Readmissions
Medical 22.5%
[22.40-22.67%]
22.4%
[22.50-24.34%]
-0.89%
[-1.82-0.05%] 0.06
Surgical 13.3%
[13.04-13.54%]
14.3%
[12.16-16.44%]
-1.01%
[-3.16-1.13%] 0.35
-
19
aAdjusted for patient and hospital characteristics and HRR fixed effects. bA p-value less than 0.0125 was considered significant.
Table 7. 30-Day Risk-Adjusteda Mortality and 30-Day Risk-Adjusted
a Readmission Rates for Hospitals
with an ICU Accredited by TJC vs. non-TJC (2015 Medicare data)
TJC
% [95% CI] Non-TJC
% [95% CI] Difference
% [95% CI] p-value
b
30-Day Mortality
Medical 8.6%
[8.54-8.70%] 8.4%
[8.17-8.72]
0.18%
[-0.11-0.47%] 0.22
Surgical 1.7%
[1.61-1.72%] 1.7%
[1.51-1.90%]
-0.04%
[-0.25-0.18%] 0.72
30-Day Readmissions
Medical 22.5%
[22.37-22.66%] 22.6%
[22.14-23.01%]
-0.07%
[-0.54-0.40%] 0.78
Surgical 13.3%
[13.04-13.57%] 13.1%
[12.24-14.02%]
0.17%
[-0.79-1.13%] 0.72
aAdjusted for patient and hospital characteristics and HRR fixed effects. bA p-value less than 0.0125 was considered significant.
Results:
The results section would benefit from more elaboration on the absolute differences in outcomes
between exposure groups and the stratified analyses mentioned above.
Thank you for this suggestion. We have now added columns to our tables of the differences in the
outcomes.
Discussion:
Regarding line 14 page 15 “given that…” to my understanding, the results are adjusted for age, sex and
so forth. Please clarify, how patients’ age and economic status adds into this argument? Furthermore,
there is no evidence in the study to that patients in the state survey hospitals are poorer – if this is the
case, it must be included in the multivariable models too. Furthermore, the claim regarding cost-
effectiveness of accreditation is not substantiated by data presented in the manuscript. The authors
should avoid making conclusions beyond presented data.
The reviewer is correct that we have adjusted our results for patient and hospital characteristics. We
agree with the reviewer about cost-effectiveness and have now deleted these sentences from our
manuscript.
Line 31 page 15 “prior work..” the authors must provide a more nuanced perspective on this matter, as
other studies have shown that providing patients with the relevant processes of care improves their
outcomes, including 30-day mortality – here are some of the work from Denmark in favor of processes on
outcomes.
Kristensen PK et al, ”Are process performance measures associated with clinical outcomes
among patients with hip fractures? A population-based cohort study.” Int J Qual Health Care. 2016 Dec
1;28(6):698-708
-
20
Ingeman A et al, “Quality of care and mortality among patients with stroke: a nationwide follow-
up study.” Med Care. 2008 Jan;46(1):63-9.
We agree and have now made changes to our manuscript to point out that the data of whether
improved process measures lead to better outcomes has shown mixed results. This is now both in the
introduction (as mentioned above) and the discussion. We had added these additional references to
our paper.
“Prior work has shown that efforts at improving clinical processes of care do not always can
lead to better patient outcomes,51,52
but this does not always hold true.53,54
Furthermore,
general hospital accreditation has demonstrated mixed results on patient outcomes.1,30,32,55
”
Line 56 page 15 “we assumed..” Please provide data or references to support the argument that TJC-
accredited hospitals are the most financially well-resourced and in addition, that finances alone can
contribute to improved patient outcomes.
Thank you for this comment. We agree that our discussion could use more clarification and a stronger
basis with references. As discussed in the introduction, TJC accreditation is expensive, not only due to
the direct costs but also the indirect costs. We are hypothesizing that hospitals with the resources and
capital to spend on hospital accreditation would not only sought out accreditation from the “gold
standard” TJC, but also had the resources and capital to direct toward improving patient safety and
quality of care. We have updated our discussion to clarify that this is our hypothesis on TJC-accredited
hospitals.
“We hypothesized that the most financially well-resourced hospitals in the country may have
sought out TJC accreditation – and since those same hospitals may have resources and capital
to dedicate toward improving patient care and outcomes, they might have better outcomes
(whether due to TJC accreditation or not). In our study, we did not find an association
between accreditation status and patient outcomes.”
Line 38 page 16 “our work..” this section need to include a discussion of how the presented result can be
compared to the referenced studies (reference 15 and 30). A correction, reference 30 includes data from
31 hospitals. If reference 30 is discussed it could be relevant to discuss another study on acute
readmissions from the same study group. In addition, the relevance of discussing papers on process
performances measures need to be explained (reference 31 and 22).
Thank you for this comment. We agree that this section could use enhancement of how our results
compared to the prior literature. To this end, we have updated our discussion. Furthermore, we have
also corrected reference 30 and added the study on readmissions from the same group (Falstie-Jensen
et al).
“Our work adds to a limited and mixed body of evidence on accreditation and outcomes.
Griffith et al. found that lower quality scores in US acute care facilities were associated with
higher mortality rates,15
while Falstie-Jensen et al found that in 31 Danish hospitals, full
accreditation – versus partial accreditation – was associated with lower mortality rates and
shorter length of stay but no difference in acute readmissions.57,58
Schmaltz and colleagues
have shown that accredited hospitals have marginally better scores on process measures,35
but
Bogh et al. found that accreditation was not associated with larger improvement for patients
-
21
with acute stroke, heart failure, or ulcers.41
Furthermore, there has been wide variation in
mortality reported between top and bottom decile accredited hospitals.59
Barnett and
colleagues found that unannounced accreditation visits by TJC lead to improved mortality for
patients admitted during that week but that the mortality rates then returned back to
baseline.60
These studies provide important information on how accreditation is associated
with process and outcomes across the globe. There is less information about how accreditation
in general, and specifically by the TJC, impacts patient outcomes across common medical and
surgical conditions as well as patient experience over a longer period of time.”
Line 17 page 17 second paragraph: the section would be stronger using fewer assumptions and more
facts including a clear discussion of the risk of selection and information bias and confounding related to
the study.
Thank you for the comment. We agree that this section could use exactly the modifications suggested
and have included a clear discussion of the risk of selection and information bias and confounding
related to the study.
“Our study has several limitations. First, as this is an observational study, we cannot assess
causality. Due to the non-randomized study design, we cannot exclude the possibility that our
results may be influenced by confounding by unmeasured factors. There is also the possibility
of selection bias of which hospitals decide to undergo accreditation. Given that accreditation
is a choice, one would assume that the institutions with better and more resources would
undergo this process, potentially biasing us towards finding a benefit of accreditation. Second,
our mortality and readmissions measures are calculated only on the Medicare population, and
we do not know if our findings are similar among commercially-insured or other publicly-
insured populations. The rates are also based on administrative claims data, which lacks
detailed clinical information. However, CMS depends on this data to assign rankings and
distribute payment determinations in national pay-for-performance programs. Moreover,
when examining patient experience, HCAHPS survey has a low response rate, and responses
are subjective and affected by personal and cultural expectations. Finally, our exposure in this
study is whether or not hospitals were accredited, and if so, by which accrediting body. Risk of
information bias is minimized due the use of national databases. We did not take into account
when the accreditation or review occurred in the last three years, as this is a cross-sectional
study.”
Conclusion:
Accreditation by TJC is perhaps the main choice but this does not automatic make it a gold standard.
Please explain how the study results substantiates the last sentence.
We completely agree with the reviewer that TJC accreditation should not automatically be considered
the gold standard without evidence. We have now reworded our sentence to the following:
Further, we found that accreditation by TJC, which is the most common form of hospital
accreditation, appears to confer no additional benefit for patients over other lesser known
accrediting agencies.