risk communication: an introduction baruch fischhoff university professor carnegie mellon department...
TRANSCRIPT
![Page 1: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/1.jpg)
Risk Communication:An Introduction
Baruch FischhoffUniversity Professor
Carnegie MellonDepartment of Social & Decision SciencesDepartment of Engineering & Public Policy
Adapted for the SuperCourse from a talk to
International Society for Pharmacoepidemiology”April 19-20, 2004
![Page 2: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/2.jpg)
Overview
Communication research strategiesTwo case studies
medical informed consentpublic health information management
Six brief examplesCommunication research management
![Page 3: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/3.jpg)
Pressure for Lay Participationin Decisions Involving Risks
healthfinanceenvironmentnational security…
![Page 4: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/4.jpg)
Poor Execution Can…
undermine effective lay decision making create feelings of helplessness erode public faith in authorities erode authorities’ faith in the public erode the social coordination produced by sharing trusted information sources
![Page 5: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/5.jpg)
Research Strategy Choices
Begin with formal analysis?Persuasive or non-persuasive?Optimality or mastery?Which individual differences?
![Page 6: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/6.jpg)
Research Strategy Choices
Begin with formal analysis?Persuasive or non-persuasive?Optimality or mastery?Which individual differences?
Also: Do you consider affect? social context? resources? …(potentially important, but outside present talk)
![Page 7: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/7.jpg)
Behavioral Decision Research
Normative analysis of decision situationDescriptive behavioral research, in terms
comparable to normative analysisPrescriptive interventions, closing gaps
between normative ideal anddescriptive reality
![Page 8: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/8.jpg)
Behavioral Decision Research Strategy Choices
Begin with formal analysis? yesPersuasive or non-persuasive? Non (default)Optimality or mastery? eitherWhich individual differences? circumstances, values, competence
![Page 9: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/9.jpg)
Three Classes of Information
QuantitativeHow big are the risks - and benefits?
QualitativeWhat determines risks - and benefits?
Communication processWhat is social context of message?
![Page 10: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/10.jpg)
calculating expected utilityassessing the value of new information setting prioritiesapplying a threshold for concern
Uses of Quantitative Information
![Page 11: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/11.jpg)
making quantitative information credible
reconciling competing claims
generating options
monitoring environment for changes
feeling realistically empowered
Uses of Qualitative Information
![Page 12: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/12.jpg)
establishing source credibility(competence, honesty)
assessing source expectations(audience competence, locus of
control)identifying conversational context
(interpretation, reading between the lines)
Uses of Process Information
![Page 13: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/13.jpg)
Surgery can extend life, but lots can go wrong.
Patients face flood of information at a stressful time.
Physicians face duty to inform.
(with Jon Merz, Paul Fischbeck, Dennis Mazur)
A Quantitative Information Example:
Carotid Endarterechtomy
![Page 14: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/14.jpg)
Normative Analysis
Value-of-information analysis, using risk analyses of patient - relevant outcomes, formalizing the materiality standard
![Page 15: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/15.jpg)
deathstrokefacial paralysismyocardial infarctionlung damageheadache resurgerytracheostomygastrointestinal upsetbroken teeth…….
Many Possible Side Effects
![Page 16: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/16.jpg)
death 15.0%stroke 5.0facial paralysis 3.0myocardial infarction 1.1lung damage 0.9headache 0.8resurgery 0.4tracheostomy 0.2gastrointestinal upset 0.09broken teeth 0.01
(% that would decline, if they knew of each risk)
But knowledge of only a few would affect many patients’ choices
![Page 17: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/17.jpg)
Descriptive Research (anticipated)
Patients don’t know probabilities of top risks: death, stroke, facial nerve paralysis
Patients don’t know event meaning forfacial paralysis
![Page 18: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/18.jpg)
verbal quantifiers (e.g., “likely” threat)experientially unfamiliar eventsvalue uncertainty (what do I really want?)
but not
very low probabilitiescumulative risk (from repeated exposure)
Prescriptive Design Issues (1)Identify Communication Challenges
![Page 19: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/19.jpg)
knowledgeinferential abilityappropriateness of confidenceappropriateness of self-efficacypersonally rational choicessatisfaction (?)
Prescriptive Design Issues (2)Choose Performance Measures
![Page 20: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/20.jpg)
Conclusions
Systematically using clinical trial results can: drastically simplify task
(for patient & physician)formalize materiality standardset research priorities
The adequacy of non-persuasive communication
is plausible, but not proven
![Page 21: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/21.jpg)
Research Strategy
Begin with formal analysis? yesPersuasive or non-persuasive? nonOptimality or mastery? ptimalityWhich individual differences?
health status, values(assume competence)
![Page 22: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/22.jpg)
A Qualitative Information (and Communication Process)
Example: Emergency Notification for Waterborne Pathogens
Cryptosporidium intrusion in domestic water supplies. Special vulnerabilityfor immunocompromised individuals.
(Liz Casman, Felicia Wu, Claire Palmgren, Mitch Small, Joan Rose, Hadi Dowlatabadi)
![Page 23: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/23.jpg)
Normative Analysis
Behaviorally realistic model of system performance, including detection, organizational coordination, and consumer response
![Page 24: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/24.jpg)
![Page 25: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/25.jpg)
Units: Averting_behavior
Definition: if consumer_awareness =0 then 0 else if consumer_awareness =1 then 1 else if info_sources > 0 then 2 else 2
Inputs: Consumer_a…Consumer Awareness for Public Systems
Info_sources Info Sources
Outputs: Consumptio…Consumption of Treated Water
Decision
Title: Averting Behavior for Public Systems
Description: Do consumers do something to avoid any possible risk of cryptosporidial infection? Correct averting behavior includes boiling drinking water and switching to safe water sources. Washing dishes, tooth brushing, and rinsing vegetables are not presently considered high risk behaviors for immunocompetent people in developed countries. Showering is not risky. Only filters with an absolute (not nominal) pore size Š 1 micron can effectively remove oocysts. (MMWR, 1995) Use of other types of filters do not constitute correct averting behavior. reference: MMWR 1995. Assessing the public health threat associated with waterborne cryptosporidiosis: report of a workshop. Rep. 44(RR-6):1-19. 0 = no action or inappropriate action (eg charcoal filter) 1 = avoid most tap water 2 = boil drinking water or use clean bottled water
![Page 26: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/26.jpg)
Descriptive Research
“Mental models” interviews (open-ended, structured by model)
with:
HIV+ individuals:knew a lot, could do little
residents of communities with past intrusions:
knew little, mildly curious
![Page 27: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/27.jpg)
Conclusions
Abandon emergency communication for crypto (perhaps OK for other contaminants, E. coli, bioterror)Provide bottled water to those at riskReevaluate communications for other immunocompromised individuals
![Page 28: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/28.jpg)
Research Strategy
Begin with formal analysis? yesPersuasive or non-persuasive? persuasiveOptimality or mastery? masteryWhich individual differences?
health status, values, competence
![Page 29: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/29.jpg)
Six Brief Examples
Sexual assault prevention adviceTeen STI riskVaccines (anthrax, MMR)Dietary supplementsHazardous chemicalsHomeland security
![Page 30: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/30.jpg)
Problem: confident, universal, contradictory advice fosters confusion and guilt
Normative: inventory of strategies, meta-analysis of effectiveness studies, identification of valuesDescriptive: nuanced belief structure, exaggerated effectivenessPrescriptive: create realistic expectations, promote research evaluating effectiveness
Sexual Assault
![Page 31: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/31.jpg)
Problem: flood of repetitious information creates illusion of understanding.
Normative: influence diagram summarizing decision-relevant scienceDescriptive: broad knowledge of HIV/AIDS, with critical “bugs”; little knowledge of other STIs, limited feeling of controlPrescriptive: targeted HIV/AIDS messages; DVD on sexual decision making
STIs (teens)
![Page 32: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/32.jpg)
Problem: mistrust of officials affects trust in medicine
Normative: comprehensive model of factors determining health, trustDescriptive: critical issues outside official problem space, terminological confusionPrescriptive: provide context, including “irrelevant concerns”; research as communication
Vaccination (anthrax, MMR)
![Page 33: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/33.jpg)
Problem: commercial-freedom-of-speech policies expanding direct-to-consumer communication
Normative analysis: model sensitive to burden of responsibility for consumer/market failureDescriptive analysis: court-mandated labels may erode consumer rationalityPrescriptive analysis: standardized, validated labels; legal standard for adequacy
Dietary supplements
![Page 34: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/34.jpg)
Problem: Can voluntary controls substitute for regulatory controls?
Normative: diffusion/uptake model, including effects of behaviorDescriptive: ineffective measures intuitively appealing (e.g., gloves, in-room breaks)Prescriptive: standardized, validated labels; legal standard for adequacy
Hazardous Chemicals
![Page 35: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/35.jpg)
Problem: Plans without behavioral input
Normative: Behaviorally realistic emergency response modelDescriptive: Simple facts not transmitted; unfounded belief in panic; specific fears decline, general anxiety stablePrescriptive: Communication to experts, interdisciplinary teams, preemptive messages
Homeland Security
![Page 36: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/36.jpg)
Also…
BSESARSHRTGMOsXenotransplantationsmallpox (vaccination)…
![Page 37: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/37.jpg)
General Conclusions
Effective risk communication requires continuing collaboration among domain specialists, risk analysts, risk communicators, practitioners, andpatients (or their representatives) -- with each willing to have its assumptions challenged.
![Page 38: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/38.jpg)
Why doesn’t everyone adoptbehavioral decision research?
![Page 39: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/39.jpg)
Possible Professional Reasons
need for specializationseparation of research worldsisolation of researchers from practitionerspredisposition to persuasionpredisposition to sweeping generalizations…
![Page 40: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/40.jpg)
Possible Management Reasons
focus on single outcomenarrow reviewing of proposals and productsfavor method application over developmentmandate to persuadedifficulty of evaluating non-persuasive communications
![Page 41: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/41.jpg)
Persuasive communication can inappropriately
undermine credibility of public and experts
alienate audienceoverlook critical audience goals &
constraintsimpose bureaucratic priorities
![Page 42: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/42.jpg)
Non-persuasive communication can inappropriately
assume nonexistent abilitiesdeny people needed protectionsmuddle value issuesleave wrong priorities unchallenged
![Page 43: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/43.jpg)
Non-persuasive communicationis needed when:
No single, clear optimal choicemultiple values, situationslarge uncertainties
No clear advisory roleexperts distrustedshared decision-making
option desired
![Page 44: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/44.jpg)
The Empirical Question
How competent are lay people to participate?
![Page 45: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/45.jpg)
Many Strong Opinions, Drawn from
assertionselected research evidenceanecdotal observation
small, unrepresentative samplesspeculative interpretationambiguity regarding decision frame
![Page 46: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/46.jpg)
technocratic control
paternalistic regulation
irrational public
free marketspopular democracy
hyper-rational public
“conservative”“liberal” Political predisposition:
Behavioral assumption:
Common Rhetorical Answers
![Page 47: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/47.jpg)
The Empirical Question(reformulated)
How competent are lay people to participate?
-- when facing specific decisions
-- and given half a chance
![Page 48: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/48.jpg)
A (Complex) Working Hypothesis
People often do sensible things if:
They get relevant information in a concise, credible form with adequate context, and without distractions
They have control over their environment and are judged by their own goals
They have some minimal decision-making competence
![Page 49: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/49.jpg)
Lita Furby, Marcia Morgan, Julie Downs, Wändi Bruine de Bruin, Sara Eggers, Matt Dombroski, Donna Riley, Mitch Small, Steve Risotto, Dan Kovacs, Paul Fischbeck, Claire Palmgren, Robyn Dawes, Rebecca Parkin, Gordon Butte, Sarah Thorne, Martha Embrey, Jenn Lerner, Roxana Gonzalez, Deb Small, Gabriel Silverman, Lisa Schwartz, Steve Woloshin…
Collaborators
![Page 50: Risk Communication: An Introduction Baruch Fischhoff University Professor Carnegie Mellon Department of Social & Decision Sciences Department of Engineering](https://reader036.vdocuments.net/reader036/viewer/2022062712/56649c905503460f94949eee/html5/thumbnails/50.jpg)
Eggers, S.L., & Fischhoff, B. (in press). A defensible claim? Behaviorally realistic evaluation standards. Journal of Public Policy and Marketing.
Fischhoff, B. (1992). Giving advice: Decision theory perspectives on sexual assault. American Psychologist, 47, 577-588.
Fischhoff, B. (1994). What forecasts (seem to) mean. Intl Journal of Forecasting, 10, 387-403.
Fischhoff, B. (1995). Risk perception and communication unplugged: Twenty years of process. Risk Analysis, 15, 137-145.
Fischhoff, B. (1999). Why (cancer) risk communication can be hard. Journal of the National Cancer Institute Monographs, 25, 7-13.
Fischhoff, B. (2000). Scientific management of science? Policy Sciences, 33, 73-87.
Sources