![Page 1: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/1.jpg)
Adaptive Models in Quantitative Research
Presented to:
QWAFAFEW – NYC Chapter
Philip Bennett Quantitative Strategist
Deutsche Insurance Asset Management
June 25, 2012
![Page 2: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/2.jpg)
A quant’s journey into the pharma sector
Areas of exploration • Drug-approval processes
• Formulary
• SMC (Scottish Medicines Consortium)
• HTA (Health technology assessment)
• OR/EMB
• QUALYs (Quality-adjusted life years)
Purpose • Collect non-typical data
• Model committee-based decision-making
• Interpret findings and determine how, when and where to use it
![Page 3: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/3.jpg)
An Exploration of the Use of Risk Management in HTA
Philip Bennett formerly: President, ETC Inc. and Director, CIBC World Markets
Lis Cook formerly: Country Manager for Scotland & Northern Ireland, Pfizer UK
Background: Original presentation of research
![Page 4: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/4.jpg)
‘Decision guidance’
• Motivation for quantitative work on the topic • Overview of the drug approval process • Collecting the data • The ‘Decision Guidance’ Model • The results • The potential uses of the model
And last, but not least . . .
• Is there anything that can be utilized in finance?
An Exploration of the Use of Risk Management in HTA
![Page 5: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/5.jpg)
• Regulatory demands Both approval requirements (clinical & HTA) have become tougher in the past
• Clinical trials The costs of trials is skyrocketing
• Budgetary pressures The need for fiscal reform conflicts with the growing need for spending on new drugs
•
Pharma: A sector facing many challenges
Quant analysis can uncover solutions for these challenges!
![Page 6: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/6.jpg)
New drugs: The economic challenge
Revenues don’t start until ALL approvals are done!
0
2000000
4000000
6000000
8000000
10000000
12000000
14000000
16000000
18000000
20000000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59
Cash Sales of Various Drugs Since Month 1 Approval
Better management of approval process can improve $$!
![Page 7: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/7.jpg)
Decision guidance: Improving the odds • Goal
Develop an analytical method with the potential to increase a product’s lifecycle revenue by ensuring the optimum chance of a successful Scottish Medicines Consortium submission outcome
• Methodology Advanced modeling techniques are used to capitalize on the informational value offered by quantitative and qualitative data and to ensure that relevant influential data is collected in a consistent, comprehensive and objective manner
• Benefit Indirectly identify value-added trade-offs between elements of the submission, facilitating effective direction and allocation of resources
![Page 8: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/8.jpg)
Decision guidance: Unique features
• Top-to-bottom design to minimize ‘entropy’/uncertainty
• Simultaneous modeling of all 12 drug categories
• Qualitatively-based input template for input consistency
• Both demand and supply factors are included
• Two-dimensional ensemble structure for both input factors and output decision outcomes
• Flagging of inconsistencies – i.e., the model tells you when it’s confused!
•
![Page 9: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/9.jpg)
Data collection: Template design is crucial
• 75 input variables Most are discrete and either ordinal (high/med/low) or categorical (drug type, manufacturer)
• 225 resulting variables via combinations (before applying dimension reduction techniques)
• Expertise required for input We tested >12 people with varying degrees of domain expertise and found < 1% variation in input values
Data input template
![Page 10: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/10.jpg)
Data analysis: Collection methodology
Variables defined according to SMC decision criteria • Hard data – comparators, formulations, budgets • Soft data – wider health service impact, trials
Subjective variables defined in terms of attributes • E.g., innovative, competitive, ‘me too’, resources attributes within wider health service impact
Attributes defined in terms of qualitative categories • E.g., positive/neutral/negative, high/moderate/low
An easy-to-use, pre-defined template providing comprehensive, consistent and objective data
![Page 11: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/11.jpg)
Data analysis: Approval factors
Unconventional soft data • Disease area • Patient group pressure • Political imperative / media
pressure • Innovation • Company credibility / litigation
experience • Assessor panel makeup • Workload
Conventional hard data • Formulations • clinical efficiency • Safety • Price (absolute and relative to
comparators) • Results versus competitor trials • Full or abbreviated submission • Budgets/political agendas
![Page 12: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/12.jpg)
Data analysis: Modeling methodology
• Based on a 40-year old economic theory that decomposes variables into their attributes
• Utilizes advances in classification, in combination with math from information theory
• Replaces classical linear approaches with ones based on non-linear dynamical systems
• Uses not one, but three techniques based on different machine learning / computational intelligence
• Results are excellent: Hybrid method maps complex input spaces to classified outcomes, such as the ‘Yes/No’ of Health Technology Assessment (HTA) decisions
![Page 13: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/13.jpg)
Overview of the model
![Page 14: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/14.jpg)
Data analysis: Modeling methodology
Two sets of models are constructed • Baseline model utilizes only annually updated data set
• Contemporaneous model utilizes additional data as available
Model design replicates expected production process • Each month’s new data serves as out-of-sample test forecast
against existing baseline and contemporaneous models • Contemporaneous model re-estimated each month
Reconciliation of results variances • Baseline model results to forecasts • Contemporaneous model results to forecasts • Baseline to contemporaneous models
![Page 15: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/15.jpg)
Data analysis: An illustration
Goal: Determine whether the raw data was consistent with our theory
• We examined just one variable and its four attributes
• We did not manipulate the data to maximize the information extracted
• We did stratify the outcomes to model ‘least’, ‘medium’ and ‘most’ successful outcomes
![Page 16: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/16.jpg)
Data analysis: Four variable attributes
Four attributes of a variable are presented in the same space
![Page 17: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/17.jpg)
Data analysis: One very complex space!
The result is complex
![Page 18: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/18.jpg)
Data analysis: An example that’s apparent
Can this least successful outcome space be explained? Fairly apparent!
![Page 19: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/19.jpg)
Data analysis: A not-so-apparent example
Can this medium successful outcome space be explained? - Not apparent!
![Page 20: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/20.jpg)
Data analysis: Not at all apparent!
Can this most successful outcome space be
explained? - Not apparent at all!
![Page 21: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/21.jpg)
‘Economic Model’ theme decision tree
![Page 22: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/22.jpg)
‘Safety’ theme decision tree
![Page 23: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/23.jpg)
‘Trials’ theme decision tree
![Page 24: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/24.jpg)
‘Manufacturers’ theme decision tree
![Page 25: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/25.jpg)
‘Cross-Theme/Ensemble’ decision tree
![Page 26: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/26.jpg)
HTA ‘Decision Guidance’: The results
In-sample Out-of-sample
Exactly correct 57 69% 17 59%
Partially correct 11 3% 5 17%
Not correct 6 7% 5 17%
Inconsistent 9 11% 2 7%
Totals 83 29
![Page 27: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/27.jpg)
Must do’s: • Non-‘elite’ drugs must include a favourable cost model
plus a demonstrable cost saving
• If not an ‘elite’ drug, trial sites must include Scotland, UK and/or North America
• To minimize chance of failure, European trials must include at least 4 sites
• To maximize chance of success, NHSS Generalizability must be at a high level
Micro-level inferences Improves submission tactics
![Page 28: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/28.jpg)
Micro-level inferences Improves submission tactics
Can do’s: • Influencing SMC meeting agenda can improve chances
of success
• Decision boundaries can be pushed by including Scottish data, RCT economic cost model, favourable cost comparators, or high trial population
• Success can be improved by including any element of high trial quality data
• Submitting ‘Elite’ indications first can improve the chances for success of the non-’elite’ indications
![Page 29: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/29.jpg)
Don’t do’s: • Don’t use low- or moderate-quality trial data if mortality is
unproven
• Don’t use trials of less than 1,000
• Don’t make submissions later than 4 months post-launch
• Don’t expect external endorsement – i.e., patient interest group or SIGN – to improve chances of a weak submission
• Don’t assume cost is primary driver as trial data, mortality and NHSS generalizability are key influencers
Micro-level inferences Improves submission tactics
![Page 30: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/30.jpg)
• Data requirements • Trial design • Trial location • Submission strategy • Launch decisions • Marketing strategy
Macro-level inferences Model leads to better overall decisions
![Page 31: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/31.jpg)
‘Decision Guidance’: The epilogue
“After being involved with and part of the SMC for over 20 years, this program can predict the Committee’s decisions as good, and in many ways, better than I can. Furthermore, this program does not have a bad day, does not go on vacations or get sick, and can actually be in more than one place at the same time.”
The sponsor of the ‘Decision Guidance’ project and the recognized expert on SMC said this at the end of the project:
![Page 32: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/32.jpg)
‘Decision Guidance’: Post-epilogue
• Expectations exceeded We exceeded the expectations of the project’s objectives…significantly
• Global applicability The potential of this applying this framework on a global basis were also proven
• Ultimately . . . . . . the dollar value of the decisions being influenced was too high and the faith in this new ‘black box’ was too low. This pharma’s decision was to ‘moth-ball’ the model!
![Page 33: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/33.jpg)
‘Decision Guidance’: Post Epilogue
Is there a place for the ‘Decision Guidance’ style of model in finance?
![Page 34: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/34.jpg)
HTA ‘Decision Guidance’
Additional Notes:
Conceptual Learning Paradigm
An Exploration of the Use of Risk Management in HTA
![Page 35: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/35.jpg)
How could we streamline our response? Raw Data Information Interpretation Application Implementation Evaluation Replication Extrapolation
![Page 36: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/36.jpg)
How could we streamline our response? Raw Data Learn Information Interpretation Application Implementation Evaluation Replication Extrapolation
![Page 37: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/37.jpg)
How could we streamline our response? Raw Data Learn Information Interpretation Application Implementation Evaluation Replication Extrapolation
![Page 38: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/38.jpg)
How could we streamline our response? Raw Data Experience Information Interpretation Application Implementation Evaluation Replication Extrapolation
![Page 39: Adaptive Models in Quantitative Research · 6/25/2012 · Health Technology Assessment (HTA) decisions . Overview of the model . Data analysis: Modeling methodology . ... • Influencing](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f3305c88551e941080d8f3d/html5/thumbnails/39.jpg)
How could we streamline our response? Raw Data Wisdom Information Interpretation Application Implementation Evaluation Replication Extrapolation
‘Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?’ – T.S. Eliot