Avai lable onl ine at www.sc iencedirect .com

journal homepage: www.elsevier .com/ locate / jva l

Can Severity Outweigh Smaller Numbers? A DeliberativePerspective from CanadaMonica Magalhaes, PhD*

Department of Global Health and Population, Harvard T.H. Chan School of Public Health, Boston, MA, USA

A B S T R A C T

Objectives: To use structured deliberation to elicit and describe thevalues of the public in Alberta, Canada, on the question of whetherthe severity of a rare condition can justify it being given priority infunding over common conditions affecting larger numbers of patients,and what aspects of a condition drive this judgment. Methods: The-matic analysis of transcripts of a group deliberative exercise carriedout as part of two citizens’ juries. The exercise was designed to elicitparticipants’ conception of disease severity, and trade-offs betweenhelping small groups with severe conditions and larger groups with lesssevere conditions. Results: In trading off severity and numbers, allgroups were willing to choose a more severely ill but smaller group ofpatients over a less severely ill but larger group of patients, althoughhow much of a severity differential was required varied between

groups. Pain that could not be relieved by alternative means was thestrongest motivator for choosing the smaller group. Other symptomswith no alternative means of relief were strong motivators as well.Conclusions: These findings indicate that, all else being equal, thepublic would support giving priority to a smaller but more severely illgroup of patients over a larger group when prioritizing the needs of thefew is life-saving, extends life enough to give hope of future improve-ment, and relieves otherwise intractable symptoms, especially pain.Keywords: citizens’ jury, deliberative methods, health policy,qualitative research, rare diseases, resource allocation.

Copyright & 2018, International Society for Pharmacoeconomics andOutcomes Research (ISPOR). Published by Elsevier Inc.

Introduction

Rare diseases are usually genetic, often chronic and severe. Theirrarity limits the potential size of the market for treatments. As aresult, absent special incentives, anticipated returns on invest-ment may be insufficient to stimulate development of treat-ments, and those that reach the market may have very highprices. To counteract these tendencies, many countries havesuccessfully adopted policies to stimulate development of treat-ments for rare diseases, help these treatments reach the market,and promote access to them [1,2]. In the absence of such a policy,by 2012, 74% of the orphan drugs approved in the United Stateswere available in the Canadian market. Canadian approvals oftenhappened later [3], and provinces adopted informal mechanismsfor providing access to these drugs [4]. In October 2012, thefederal government started to develop an orphan drug regulatoryframework.

Because drugs for rare diseases are rarely considered cost-effective by usual criteria [5], decisions to cover these drugsrequire departing from usual cost-effectiveness requirements,treating these diseases as priority cases for which society iswilling to pay more than it would pay to treat common diseases.Studies find that the public is unwilling to pay more for the samehealth gains because of a disease’s rarity [6–8], but supportsprioritizing the more severely ill, to some extent: a person

trade-off study in Norway finds no preference for severity, allelse being equal [8]. A person trade-off study in the United States[9] finds preference for severity even at the expense of someaggregate gain, whereas one in the United Kingdom [10] finds nopreference for rarity but preference for severity only if health gainis substantial. A deliberative study in the United Kingdom findsthat the “rule of rescue” was one of three rationing principleschosen by deliberators [11]. Reviews of preference studies find aconsistent preference for severity, although its strength varies[12,13]. A review of social value arguments also finds support forvaluing severity but not rarity [14].

Orphan drugs treat diseases that are not only rare, but alsosevere. Because there is no preference for rarity but there ispreference for severity, it becomes necessary to investigatewhether severity can outweigh smaller numbers in public pref-erences for funding allocation, and, if so, what aspects of severitycan do so. The exercise described here aims to elicit consideredjudgments from citizens of Alberta regarding this question, aswell as the rationales for these judgments.

Methods

Citizens’ juries are a method for engaging the public in structureddeliberation. Deliberation allows citizens to articulate and justify

1098-3015$36.00 – see front matter Copyright & 2018, International Society for Pharmacoeconomics and Outcomes Research (ISPOR).

Published by Elsevier Inc.

https://doi.org/10.1016/j.jval.2018.03.010

* Address correspondence to: Monica Magalhaes, Department of Global Health and Population, Harvard T. H. Chan School of PublicHealth, 677 Huntington Ave, Boston, MA 02115.

E-mail: mmagalhaes@hsph.harvard.edu.

V A L U E I N H E A L T H 2 1 ( 2 0 1 8 ) 5 3 2 – 5 3 7

their views in terms that should be acceptable to others, to beexposed to the views of others, and to broaden and revise theirown views [15]. Research on the effects of citizens’ juries showsthat participation increases jurors’ openness to changing theirviews, influences jurors’ opinions, and has long-term effects ontheir level of information about the policy issue in question andpositive effects on trust in health technology assessment proc-esses [16–19].

Two citizens’ juries were involved in this study. Each con-sisted of 16 members of the public who were compensated fortheir time to make participation accessible for all socioeconomicstrata. To minimize selection bias due to inability to get time offof work, juries were held on a Friday evening, Saturday, andSunday. The first jury took place from April 17 to 19, 2015, and thesecond from June 5 to 7, 2015, both in Edmonton, Alberta.

From a sampling frame provided by the postal service coveringnorthern Alberta, 3000 potential participants were randomlyselected and sent information packages about the juries. The 224package recipients who manifested a willingness to participatewere screened by telephone interviews through which researcherscollected demographic information. Health care workers, currentpatients, individuals involved in advocacy, and others with a pre-existing interest in the health care system were excluded from thepool of potential participants. From those remaining, a samplestratified by age, ethnicity, socioeconomic status, and occupationwas drawn, with random selection within groups (Table 1).

Each jury started with the testimony of expert witnesses, whoserole was to familiarize participants with issues relating to rarediseases, from various perspectives. Among the information pro-vided to the group by the witnesses, decision makers fromprovincial bodies gave an overview of the system for approvaland reimbursement of therapies in Alberta and of the currentissues they weremost concerned with at the time. A physician whotreated patients with rare diseases gave general facts about raredisease patients in Alberta and the issues they face, and theoptions open to a clinician to obtain treatments that are notusually covered. A patient with rare disease and a mother andcaregiver of such a patient both told their personal histories, whichtouched on some of the themes the clinician and decision makershad talked about. A representative of the pharmaceutical industrydiscussed the high costs and difficulties of doing research with asmall number of patients, the need to apply to health authoritiesfor coverage, and the influence of public payers on private manu-facturers’ investment decisions.

For the following day and a half, jurors were asked tocomplete a series of deliberative exercises.

For this exercise, each jury was divided into three groups, twowith five members and one with six members. Each group wasassisted by two researchers working as facilitators. First, thegroup was presented with a set of cards, each containing a healthstate profile described by five attributes, with each attributehaving three possible levels (Table 2). Profiles described real

Table 1 – Sociodemographic profile of jury participants.

Characteristic Jury 1 Jury 2 Alberta Source of Alberta data

SexMale 8 (50%) 8 (50%) 50% 2016 data [20]Female 8 (50%) 8 (50%) 50%

Age (y)18–24 2 (13%) 2 (13%) 6% (age 20–24 years) 2016 data [20]25–34 2 (13%) 2 (13%) 16%35–44 3 (19%) 3 (19%) 14%45–54 3 (19%) 3 (19%) 14%55–64 2 (13%) 2 (13%) 12%65–74 2 (13%) 2 (13%) 7%474 2 (13%) 2 (13%) 7%

Education (highest level)Less than high school 1 (6%) 1 (6%) 11% 2016 data [20]High school 7 (44%) 5 (31%) 25% Population 25–64 yearsPostsecondary diploma 4 (25%) 5 (31%) 36%Undergraduate degree 4 (25%) 3 (19%) 22%Graduate degree 0 (0%) 2 (13%) 6%

Annual household income (Can$, before taxes)o25,000 3 (19%) 2 (13%) 9% 2016 data [20]25,000–45,000 2 (13%) 4 (25%) 11% Refers to income in the year 201546,000–70,000 5 (31%) 4 (25%) 16%71,000–100,000 3 (19%) 3 (19%) 17%4100,000 3 (19%) 3 (19%) 47%

Employment statusEmployed 7 (44%) 9 (56%) 65% (employed) 2016 data [20]Unemployed 4 (25%) 3 (19%) 6% (unemployed) Population 15 years and olderRetired 5 (31%) 4 (25%) 28% (not in labor

force)EthnicityAsian 1 (6%) 1 (6%) 19% 2016 data [20]White 14 (87%) 14 (87%) 89% Total is 4100% because multiple answers were

allowedFirst Nations (Aboriginal) 1 (6%) 1 (6%) 8%Other 0 0 4%

Geographical locationUrban 11 (69%) 13 (81%) 83% 2011 data [21]Rural 5 (31%) 3 (19%) 17%

V A L U E I N H E A L T H 2 1 ( 2 0 1 8 ) 5 3 2 – 5 3 7 533

diseases, to ensure they were realistic, but participants were nottold the names of the diseases represented by the profiles tominimize bias due to experiences with particular conditions. Thegroup then ranked these diseases from most severe (1) to leastsevere (14 in the first jury, 15 in the second jury). It was importantthat jurors themselves ordered conditions by severity. Otherwise,a choice to treat a large population might indicate either thatjurors prioritize the larger number or that they consider thecondition of the larger number to be more severe.

After diseases were ranked, facilitators presented a scenarioin which the group had to choose between a large population(10,000 patients) and a small population (100 patients), both inthe health state ranked as most severe: Which population wouldthe group prioritize if only one could receive treatment? Partic-ipants were not told the expected health gains from treatment, toavoid either the assumption that the larger health gains would goto the largest number or that the larger health gains would go tothose starting at the most severe baseline. If asked, facilitatorssaid the expected gains were unknown.

The groups, as expected, selected the large population. Facilita-tors then posed the same question, iteratively, each timemoving thelarge population one step up in the ranking of conditions andmaintaining the small population at the most severe state. Thisprocess was repeated until participants “tipped,” choosing to priori-tize the small population in the most severe state over the largegroup in a less severe state. The groups’ collective tipping point wasconsidered to be the one where most chose to prioritize the smallpopulation. To capture minority views, the exercise continued untilall participants had chosen to prioritize the small population.

Of the six groups that undertook the exercise, five wererecorded and transcribed for thematic analysis. Transcripts were

entered into NVivo software (QSR International, London). Codingstarted with a descriptive codebook, with categories correspond-ing to the attributes of health states (“life expectancy,” “pain anddiscomfort,” etc.). During the first round of coding, new codeswere added corresponding to recurring themes (e.g., “hope,”“alternative symptom relief,” and “activities of daily living/inde-pendence”). Subsequent rounds of coding identified instances ofthese emerging themes throughout the data. This allowed prev-alent perceptions of each attribute to be identified, as well asoverarching themes that cut across attributes and therefore mayreflect principles or values that consistently matter to jurors.

Results

All groups arrived at similar severity rankings of the diseaseprofiles. Five of the six groups tipped from treating the largepopulation to treating the small population at or near the mid-point of the severity spectrum. In the first jury’s three groups, thetipping points were the conditions ranked at number 5, 6, and 7 inthe 1 to 14 spectrum. In the second jury’s three groups, the tippingpoints ranked at number 7, 7, and 3 in the 1 to 15 spectrum.

Groups’ Views on Each Attribute

When deliberating about which conditions are more severe,jurors’ main strategy was to compare the subjective experienceof being in each condition, putting themselves “in the shoes” ofthe patients.

As might be expected given the focus on subjective experi-ence, “pain and discomfort” was the most salient dimension injurors’ assessment of severity:

4: So pain and discomfort. That’s—the main goal of medicine, or oneof them, is to alleviate pain.3: We want to prolong life, but not in pain.5: I would choose to [live] less but better. So probably we have todecide how many people are for—1: Longer life with more pain or less life with less pain.4: Less life with less pain. (Group 3)

Similarly, “depression and anxiety,” by definition an experi-ential dimension of health states, received a great deal ofattention:

4: Because of the severe depression and anxiety?5: Because that’s something you’re totally aware of. (Group 2)

“Life expectancy” was in some cases valued as time forimproving one’s health state, and in other cases valued accordingto the quality of the life being lived (see the “Emphasis on thesubjective experience of hope” section).

“Physical impairment” was associated with loss of independ-ence and ability to carry out activities of daily living, and,consequently, loss of life enjoyment:

2: There’s no happiness at all in that one. Severely impaired? That’slike, just lying there. (Group 3)

There were conflicting views about what the subjectiveexperience of “cognitive impairment” would be like. Some jurorsthought that lacking awareness of one’s situation might mitigateseverity, but others thought the opposite:

3: But cognitive—severely [cognitively impaired] just means they can’tremember, they can still maybe just be happy. They just don’tremember. (Group 3)

Table 2 – Attributes and levels of health states.

Jury Attribute Level

Jury 1 Life expectancy o1 year1–5 years45 years

Physical functioning Severely impairedModerately impairedMildly impaired

Cognitive functioning Severely impairedModerately impairedMildly impaired

Pain and discomfort ExtremeModerateMild

Depression and anxiety SevereModerateMild

Jury 2 Life expectancy o1 year1–5 years45 years

Ability to function Severely impairedModerately impairedMildly impaired

Pain and discomfort ExtremeModerateMild

Depression and anxiety SevereModerateMild

Caregiver burden HighMediumLow

V A L U E I N H E A L T H 2 1 ( 2 0 1 8 ) 5 3 2 – 5 3 7534

1: Maybe that’s why you’re depressed. Because you don’t have anyidea what’s going on.2: No, that’s actually the opposite.5: Yeah, that’s the opposite. (Group 2)

“Caregiver burden” was discussed not only in the second jury,when it was an attribute of the health states described in thecards, but also in the first jury, when it was introduced to thediscussion by participants with no prompting. Jurors consideredthe impact of high caregiver burden on families, but did not seemto include it in the concept of severity. In keeping with the focuson the patients’ experience, one theme that came up in connec-tion with caregiver burden was the experience that the patientmight have of feeling like a burden to others, especially in thecontext of severe physical impairment that leaves the patient inneed of constant assistance for personal care and daily activities.

1: … But say you’re severely [physically impaired] and you can’t donothing but only for one to five years. As opposed to somebody who’sgoing through their day and now you’re incapable of not doing someof your daily—you feel like you’re a burden. And it’s going to makeyour depression worse over time. It is treatable, and you have time,but. (Group 2)

Cross-Attribute Patterns in Ranking Severity

Discounting of symptoms that can be relieved without treatingthe illnessJurors reduced the severity attributed to health states when therewas a possibility of relieving symptoms, even if relieving thesymptoms did not treat the disease or the condition itself.Instances of this pattern are described in this section.

In the handouts containing the descriptions of the attributesand levels of health states (see Supplemental Materials foundat https://doi.org/10.1016/j.jval.2018.03.010), jurors were told thatthe mild and moderate levels of pain could be controlled withdrugs, whereas severe pain could not. As a result, mildpain and moderate pain were treated interchangeably and notgiven much weight in attributing severity. This occurred in allgroups.

4: If [depression] is treatable, then why are we putting the mild oneway up there [at the severe end of the spectrum]?5: Because they’ve got the extreme pain. That’s not treatable, right?See this one—that’s what I’m saying, this is treatable, but the severe—4: So we’re saying pain is a higher priority.2: If it’s moderate and it’s mild, it’s reasonable, right? It’s treatable.5: You can take a painkiller. But the extreme people can’t takepainkillers. They don’t work. (Group 2)

Jurors were not told whether depression and anxiety at anylevel could be relieved by drugs. Nevertheless, some groups madethe reasonable assumption that psychiatric treatment couldrelieve this symptom, and therefore also discounted depressionand anxiety when judging the severity of health states containingdepression.

4: I’d say depression’s treatable, myself. Because I’ve been treated, so.… You can’t treat the pain, but you can treat depression. …1: I understand depression is bad, but there’s a way to treat it.(Group 2)

One group saw caregiver burden as a “symptom” that could berelieved in ways other than by treating the disease. This groupthen decided to discount caregiver burden in the disease severityrankings for this reason.

5: … The caregiver thing is not unimportant, but they’re … thecaregiver has an opportunity or potential opportunity to relieve him orherself from that role for some period of time. Not always, but there isthat potential …. (Group 5)

Emphasis on the subjective experience of hopeAppeals to hope were a recurring feature of the deliberations.Hope was characterized as a belief or subjective feeling that aperson in the health state being discussed might experience, thathowever bad their experience is, it can get better in the future.The absence of subjective hope led jurors to rank a disease asmore severe. Some instances of this pattern are described.

Depression, when understood in a way that emphasized hope-lessness about the future, carried more weight than it did when thediscussion emphasized the experience of low mood or lack of lifeenjoyment. The latter view of depression was given less weight inthe attribution of severity than other attributes of the health states.

3: Yeah. But with these [severely depressed] too I say there is hope.4: Well not in their perception, no.3: Not in their perception, no. (Group 5)

Severe pain that cannot be relieved, besides being consideredthe worst attribute from an experiential point of view, was alsoviewed as a hopeless state to be in because of there being noperspective of relief.

2: Because this is worse, for me, myself. Like, I’d rather be indepression than … be in pain and not [able] to function, becauseyou’re in pain and you can’t do anything about it. And that sucksrather than just not enjoying life enough. (Group 6)

Longer life expectancy was considered to reduce severity, aswould be expected, when it was understood as more time topotentially improve one’s health state by receiving treatment orby new treatments becoming available.

1: In five years, who knows what other drugs are coming out. (Group 3)

More surprisingly, longer life expectancy was judged toincrease the severity of a condition if it was seen as prolonginga hopeless state. Several jurors held this view, and one articu-lated this point particularly forcefully:

2: This person’s been sentenced to one to five. This one’s beensentenced to under one. (Group 3)

These two views of life expectancy were a source of disagree-ment between jurors.

Trading Off Severity and Numbers

When deliberating about whether to prioritize a small and moreseverely ill population (100 patients) or a large and less severely illpopulation (10,000 patients), jurors began by choosing the largernumber. This was expected, because in the first few questions thelarge and small populations were both in severe states of health.As the large number of patients was moved to less severeconditions, however, the presumption of priority to the largerpopulation weakened. Jurors expressed a willingness to sacrificenumbers for the sake of prioritizing the worst off, to some extent.

5: Well, obviously, if we continue going down this way, we’re going tosacrifice 100 every time.3: No, I think we might allow some people to live with some kind ofthing and save these guys.5: Just because the condition is so bad. (Group 3)

V A L U E I N H E A L T H 2 1 ( 2 0 1 8 ) 5 3 2 – 5 3 7 535

Subjective experience continued to be a primary considerationin trade-off discussions, and intractable pain continued to be thefactor most prominently taken into account. Few participantschose the small population when the large population hadintractable pain. Among all other participants, tipping towardprioritizing the small group began as soon as the large populationno longer had extreme pain and would subsequently only be inhealth states containing treatable (mild or medium) pain.

Facilitator: At this point you would switch [to the smaller population],and why is that? Why would you switch at this point?2: Because these people are severely impaired at this point. And thepain and discomfort is extreme. At this point I’m assuming that withmild pain and discomfort [for the larger population], it could becontrolled. (Group 4)

After the first participants began tipping, there were explicitdisagreements about whether two conditions were differentenough in severity to justify leaving the large population untreated:

2: I would argue because those life expectancies [of the largepopulation, 1 to 5 years; and the small population, less than 1 year]are fairly close, like less than a year could be 11 months, and 1 to 5years can be 1 year. So like, if it was over 5 years then I will be morelenient, like 1 to 5 years can be fairly close together. … I think it comesout to the quantity of people again. (Group 6)

Ability to relieve symptoms also continued to be an importantconsideration:

2: These people [small population with more severe condition] have alow quality of life because they’re in pain and cannot get relieved withdrugs … versus this group, I know there’s more of them, but theyhave some relief and so …. (Group 5)

The idea that longer life expectancy may be bad for thepatient did not arise in the trade-off stage. Jurors often lookedat life expectancy as an indicator of urgency, counting in favor ofgiving a condition increased priority. Patients who were expectedto live longer could wait.

5: And with this [large] group you have time. … So I will save these100 first. (Group 6)

Throughout the priority setting deliberations, participantsexpressed a deepening appreciation for the difficulty of thechoices faced by decision makers.

2: So you’re allowing 10,000 to ….3: Cold, calloused. What can I say? (Group 2)

3: It’s trying to balance … do you go for more or, like, when do youhave compassion for ….

1: The hundred people. (Group 3)

2: I can see how difficult this gets. That’s so tough. (Group 6)

Discussion

When a health state combined longer life expectancy withintractable pain, groups viewed the condition as extremelyundesirable and therefore severe—even more severe than ahealth state containing extreme pain and shorter life expectancy.This explains decisions that could be thought of as wrong or

irrational: three of the six groups picked as the most severecondition a health profile that strictly dominates another profile,which the group ranked as less severe. These choices do notappear to be mistakes on the part of the jurors. There is noevidence of misunderstanding of the health states or of theranking task in the transcripts. Rather, these choices appear toindicate that the concept of severity used by the groups refersonly to current quality of life and does not include lifeexpectancy.

The idea that a longer life expectancy with very low quality oflife may be bad for the patient was present at the ranking stagebut not at the priority setting stage. This appears to be becausewhen there is a possibility of treatment, longer life is no longer a“sentence” to poor quality of life. Therefore, both patient popula-tions could be assumed to want to receive treatment, and thosewith less time to live if untreated have the most pressing need. Inother words, shorter life expectancy, which in the first stage issometimes counted as a reason to rank a condition as less severethan an otherwise similar condition with a longer life expectancy,now counts in favor of giving it more priority. Jurors’ under-standing of severity as excluding life expectancy, therefore, is aninteresting observation but has no implications for jurors’ prioritysetting choices.

Intractable pain stood out as a particularly strong motivatorfor tipping, or choosing to prioritize a population of 100 patientsover a population of 10,000 patients. Four of the six groups rankedthe four profiles containing extreme pain (that cannot be relievedwith medication) as the four most severe. This was to beexpected, given the correlations between dimensions of diseaseprofiles (i.e., more pain tracks more physical impairment andhigher caregiver burden). Nevertheless, pain was usually thedimension that was mentioned as justification for prioritizingthe small population. Strong disagreement over whether to “tip”toward the small population usually started when the largepopulation no longer had extreme pain, whereas the smallpopulation, remaining constant at the most severe condition, did.

The values expressed by the participants have the followingimplications that are relevant to the problem of rare diseases (allelse being equal):

1. Lack of alternative treatments or options to manage symp-toms, particularly pain, takes priority over large numbers.

2. Urgency takes priority over large numbers if the life-extendingintervention can offer hope of an acceptable quality of life in thefuture (e.g., because of new treatments becoming available).

3. Patients’ needs take priority over those of caregivers.4. The consideration of whether a treatment is disease-altering

rather than symptom management takes low priority.

Study Limitations

Sampling was geographically limited to northern Alberta, butprevious research has shown high consistency between out-comes of citizens’ juries in northern and southern Alberta [19].A limitation of the analysis is that it was performed by a singlecoder, and therefore was vulnerable to observer bias. This wasmitigated by keeping the coding as close as possible to thetranscripts, using participants’ own words as codes wheneverpossible. To minimize the scope for projection or introduction ofidiosyncratic views [22], reasons for participant’s choices werenot imputed unless they were stated explicitly, as demonstratedby the illustrative quotes provided.

Possible framing effects may have affected the results.Although the format of the exercise framed both options in eachtrade-off question in the same terms, giving the small and thelarge populations equal salience, it is possible that hearing from apatient with a rare disease and from the mother of such a patient

V A L U E I N H E A L T H 2 1 ( 2 0 1 8 ) 5 3 2 – 5 3 7536

the day before the exercise made the plight of those affected byrare diseases more vivid to participants, disposing them towardthe smaller population. The framing of the jury as an activityrelating to rare diseases may also have had this effect. Never-theless, such effects, if any, appear to have been small: caregiverburden was not prioritized, in spite of the testimony of themother of a patient with a rare disease also having the potentialof making the plight of caregivers more salient.

Because of the narrow scope of this study, which looks only atseverity of initial health state and number of patients, holding allelse equal, these findings alone do not predict or guide real-worldjudgments about treatments. As frameworks of broader scope[14] and other citizens’ juries carried out in Alberta [23,24] show,severity and numbers are relevant to resource allocation, butother factors can be equally or more important. Most notably, thisstudy did not take into account potential health gains or costs oftreatment. The abstract framing of the diseases and the ques-tions also leaves out concrete and contextual information thatwould affect real-world priority setting choices.

Conclusions

The findings of the study indicate that, all else being equal, thepublic would support giving priority to a smaller but moreseverely ill patient population over a larger patient populationwhen prioritizing the needs of the few is life-saving, extends lifeenough to give hope of future improvement, and relieves other-wise intractable symptoms, especially pain.

Acknowledgments

I am deeply indebted to Devidas Menon, Alexa Parry, and AudraVair for their input during the creation of the exercise and forfacilitating group sessions, and to Kathy Swartz for valuablecomments at the writing stage. Responsibility for any errorsand interpretation of the data remains solely with me.

Source of financial support: This work was supported by aCanadian Institutes of Health Research Emerging Team grant for“Developing Effective Policies for Managing Technologies for RareDiseases” (grant no. 119194).

Supplemental Materials

Supplemental material accompanying this article can be found inthe online version as a hyperlink at http://dx.doi.org/10.1016/j.jval.2018.03.010 or, if a hard copy of article, at www.valueinhealthjournal.com/issues (select volume, issue, and article).

R E F E R E N C E S

[1] Hughes DA, Poletti-Hughes J. Profitability and market value of orphandrug companies: a retrospective, propensity-matched case-controlstudy. PLoS One 2016;11:1–12.

[2] Haffner ME. Adopting orphan drugs—two dozen years of treating rarediseases. N Engl J Med 2006;354:445–7.

[3] Herder M, Krahn TM. Some numbers behind Canada’s decision to adoptan orphan drug policy: US orphan drug approvals in Canada, 1997–2012.Health Policy 2016;11:70–81.

[4] Menon D, Clark D, Stafinski T. Reimbursement of drugs for rarediseases through the public healthcare system in Canada: Where arewe now? Healthc Policy 2015;11:15–32.

[5] Hughes DA, Tunnage B, Yeo ST. Drugs for exceptionally rare diseases:Do they deserve special status for funding? QJM 2005;98:829–36.

[6] Mentzakis E, Stefanowska P, Hurley J. A discrete choice experimentinvestigating preferences for funding drugs used to treat orphandiseases: an exploratory study. Health Econ Policy Law 2011;6:405–33.

[7] Linley WG, Hughes DA. Societal views on NICE, cancer drugsfund and value-based pricing criteria for prioritising medicines: a cross-sectional survey of 4118 adults in Great Britain. Health Econ2013;22:948–64.

[8] Desser AS, Gyrd-Hansen D, Olsen JA, et al. Societal views on orphandrugs: cross sectional survey of Norwegians aged 40 to 67. BMJ2010;341:1–6.

[9] Ubel PA. How stable are people’s preferences for giving priority toseverely ill patients? Soc Sci Decis Making 1999;49:895–903.

[10] Linley WG, Hughes DA. Societal views on NICE, cancer drugs fundand value-based pricing criteria for prioritising medicines: a cross-sectional survey of 4118 adults in Great Britain. Health Econ 2013;22:948–964.

[11] Cookson R, Dolan P. Public views on health care rationing: a groupdiscussion study. Health Policy 1999;49:63–74.

[12] Nord E, Johansen R. Concerns for severity in priority setting inhealth care: a review of trade-off data in preference studies andimplications for societal willingness to pay for a QALY. Health Policy2014;116:281–8.

[13] Shah KK. Severity of illness and priority setting in healthcare: a reviewof the literature. Health Policy 2009;93:77–84.

[14] Paulden M, Stafinski T, Menon D, McCabe C. Value-basedreimbursement decisions for orphan drugs: a scoping review anddecision framework. Pharmacoeconomics 2015;33:255–69.

[15] Fearon JD. Deliberation as discussion. In: Elster J, ed. DeliberativeDemocracy. Cambridge, UK: Cambridge University Press, 1998.

[16] Dolan P, Cookson R, Ferguson B. Effect of discussion and deliberationon the public’s views of priority setting in health care: focus groupstudy. BMJ 1999;318:916–9.

[17] Abelson J, Bombard Y, Gauvin F, et al. Assessing the impacts of citizendeliberations on the health technology process. Int J Technol AssessHealth Care 2013;29:282–9.

[18] Stafinski T, Menon D, Yasui Y. Assessing the impact of deliberativeprocesses on the views of participants: Is it “in one ear and out theother”? Health Expect 2012;17:278–90.

[19] Carman KL, Mallery C, Maurer M, et al. Effectiveness ofpublic deliberation methods for gathering input on issues inhealthcare: results from a randomized trial. Soc Sci Med 2015;133:11–20.

[20] Statistics Canada. Census profile, 2016 census (Alberta).Available from: http://www12.statcan.gc.ca/census-recensement/2016/dp-pd/prof/details/page.cfm?Lang=E&Geo1=PR&Code1=48&Geo2=PR&Code2=01&Data=Count&SearchText=48&SearchType=Begins&SearchPR=01&B1=All&Custom=&TABID=3. [Accessed January 31, 2018].

[21] Statistics Canada. Population, urban and rural, by province andterritory (Alberta). Available from: https://www.statcan.gc.ca/tables-tableaux/sum-som/l01/cst01/demo62j-eng.htm. [AccessedJanuary 31, 2018].

[22] Boyatzis RE. Transforming Qualitative Information. Thousand Oaks,CA: Sage Publications, 1998.

[23] Menon D, Stafinski T. Engaging the public in priority-setting for healthtechnology assessment: findings from a citizens’ jury. Health Expect2008;11:282–93.

[24] Stafinski T, McCabe C, Menon D. Determining social values for resourceallocation decision-making in cancer care: a Canadian experiment.J Cancer Policy 2014;2:81–8.

V A L U E I N H E A L T H 2 1 ( 2 0 1 8 ) 5 3 2 – 5 3 7 537