fema national preparedness assessment division (npad ......convenient data set that enabled hssedi...

Prepared for: Department of Homeland Security, The Federal Emergency Management Agency (FEMA)

FEMA National Preparedness Assessment Division (NPAD) Homeland Security Grant Program Investment Benefit Pilot Study Results

July 18, 2019

This document is a product of the Homeland Security Systems Engineering and Development Institute (HSSEDI™).

ii

Homeland Security Systems Engineering & Development Institute The Homeland Security Systems Engineering & Development Institute (HSSEDI) is a federally funded research and development center (FFRDC) established by the Secretary of Homeland Security under Section 305 of the Homeland Security Act of 2002. The MITRE Corporation operates HSSEDI under the Department of Homeland Security (DHS) contract number HSHQDC-14-D-00006. HSSEDI’s mission is to assist the Secretary of Homeland Security, the Under Secretary for Science and Technology, and the DHS operating elements in addressing national homeland security system development issues where technical and systems engineering expertise is required. HSSEDI also consults with other government agencies, nongovernmental organizations, institutions of higher education, and nonprofit organizations. HSSEDI delivers independent and objective analyses and advice to support systems development, decision making, alternative approaches, and new insight into significant acquisition issues. HSSEDI’s research is undertaken by mutual consent with DHS and is organized by tasks. This report presents the results of investment benefit pilot studies conducted as alternatives to the more traditional return on investment analysis conducted under Task Order No. HSFE2017J0058, entitled Federal Emergency Management Agency (FEMA) National Preparedness Assessment Division (NPAD) Homeland Security Grant Program (HSGP) Return on Investment/Cost-Benefit Analysis Methodology and Software Support Tool. The purpose of the task is to provide FEMA recommendations for effectively developing and executing a program-level return-on-investment estimate and a project-level benefit-cost analysis for its non-disaster emergency preparedness grants. The information presented in this report does not necessarily reflect official DHS opinion or policy. This document was prepared for authorized distribution only. It has not been approved for public release.

For more information about this publication contact:

Homeland Security Systems Engineering & Development Institute

The MITRE Corporation 7515 Colshire Drive McLean, VA 22102

Email: [email protected]

http://www.mitre.org/HSSEDI

mailto:[email protected]

http://www.mitre.org/HSSEDI

iii

Abstract The Federal Emergency Management Agency (FEMA) National Preparedness Assessment Division (NPAD) asked the Homeland Security Systems Engineering and Development Institute (HSSEDI) to support their efforts to understand the return on investment (ROI) provided by the FEMA non-disaster emergency preparedness grants under the Homeland Security Grant Program (HSGP). Previous work determined that the data required to perform an ROI study is not available. This report describes three pilots designed to test alternative approaches to determining the benefits of the HSGP: the Detailed Operations Model (DOM), a Revealed Preference Analysis (RPA), and the Breakeven Analysis (BA). The approaches, results, and recommendations of the exploratory pilots are presented here. The purpose of the pilots was to develop the approaches and explore the feasibility of using them at scale.

Key Words 1. FEMA NPAD 2. Investment Benefit 3. System Dynamics 4. Revealed Preference Analysis 5. Breakeven Analysis

iv

This page intentionally left blank.

v

Executive Summary The Federal Emergency Management Agency (FEMA) National Preparedness Assessment Division (NPAD) engaged the Homeland Security Systems Engineering and Development Institute (HSSEDI) to conduct a study of the return on investment (ROI) of the FEMA Homeland Security Grant Program (HSGP). As the data required to conduct the ROI study do not exist, HSSEDI proposed conducting three pilot studies to test various methodologies to determine if it was possible to arrive at the investment benefits of HSGP1 without the requisite data. This report describes the approach, results, and recommendations of those pilots: Detailed Operations Model (DOM), a Revealed Preference Analysis (RPA) of Virginia State Homeland Security Program (SHSP) grant proposals, and a Breakeven Analysis (BA).

Detailed Operations Model

Objective The DOM pilot used HSGP-funded 2015 Search and Rescue (SAR) projects in Florida to test a methodology that combined multi-attribute utility theory (MAUT) with system dynamics modeling.2 The objective was to determine if it was possible to identify the non-monetary disaster preparedness benefits that selected communities received as the result of these HSGP-funded projects. If the pilot proved successful, additional studies could be conducted to test the scalability of the approach.

Approach This pilot study used a hierarchical value tree based on MAUT to link investments, and the projects within them, to overall benefit,3 as shown in Figure 1.

1 Examples of the missing data are cause-effect relationship measures, outcomes from previous investments, and inconsistent and non-standardized investment descriptions. 2 MAUT is a structured methodology designed to handle the tradeoffs among multiple objectives. System dynamics is the recognition that the structure of any system, is often just as important in determining its behavior as the individual components themselves. 3 DOM defines benefit as reduced risk of negative consequences—lives lost, injuries/illnesses, damage, business losses, time spent without necessities, and impact on other quality of life (QOL) issues.

vi

Overall benefit

Major benefit (risk reduction) categories

THIRA targets

Investments

Figure 1. Alignment of Investments to Targets to Benefits

Simulation was used to estimate the contribution/operational impact of investments toward a Threat and Hazard Identification and Risk Assessment (THIRA) target. HSSEDI used the Mass Search and Rescue THIRA target— “Within (#) hours of an incident, conduct inland search and rescue operations to locate all (#) individuals requiring rescue across a (#) square mile area”—proposed in FEMA’s Draft THIRA standardized target list.4 The simulation output is assumed to represent the utility of the investment. Subject matter experts (SMEs) assessed the contributions of all THIRA targets toward the benefit categories. The impact score (0 = no impact; 20 = low impact; 50 = medium impact; and 80 = high impact) indicates how much each target contributes to each benefit category. The value tree was developed using the benefit category contribution assessments. For the DOM pilot, only the Search and Rescue target was assessed. The SME scores5 for the contribution toward benefit categories appear in Table 1.

Table 1. SME Scores for SAR Target

Benefit Category SME Score(s) Rationale

Reduced risk of lives lost 80 The primary goal and highest impact of search and rescue operations is saving lives.

Reduced risk of injuries 80 Rescuing people reduces injuries that may befall them if not rescued.

Other quality of life 60 When people are rescued, they may be delivered to a shelter or safe place where their life is no longer in danger and assistance provided.

Reduced risk of time without necessities 40 When people are rescued, they may be delivered to a shelter or

safe place where necessities and/or assistance are provided.

Reduced risk of property damage 23.3 This is not the primary goal or expected benefit of search and

rescue operations.

4 Draft THIRA Target Review20180112.xlsx, provided during FEMA NPAD meeting, 30 January 2018. 5 HSSEDI chose to employ a high-low-medium scale because of the highly qualitative nature of the assessments and chose values of 80, 50, and 20 to represent those three points. The SME scores shown in the table are an average of the SME scoring results.

vii

Benefit Category SME Score(s) Rationale

Reduced risk of business losses 23.3 This is not the primary goal or expected benefit of search and

rescue operations.

A weighted average rollup function reflecting the relative importance of each benefit category prescribes how the scores of all the benefit categories are combined to produce the overall “preparedness” score or overall benefit. The HSSEDI team found no reference or data elements in the information reviewed that could provide the weights for these elements. As a result, an elicitation process was conducted using HSSEDI SMEs to provide the weights shown in Table 2.

Table 2. Major Benefit Category Contribution to Overall Investment Benefit

Category Weight

Reduced risk of lives lost 1.00

Reduced risk of injuries/illnesses 0.50

Reduced risk of property damage 0.50

Reduced risk of business losses 0.50

Reduced risk of time spent without necessities 0.25

Reduced risk of impact on other QOL issues 0.25

Results Implementation of the DOM required significant use of SME input. A number of assumptions required to establish the model’s baseline and the lack of accepted points of reference for those elements remain unresolved. The HSSEDI team could not foresee any means by which output of the model could be assessed and the model tuned for consistent operation. Instead, DOM analysis results were compared among each other to produce relative valuations, as opposed to benefit values for the projects. The result of the DOM analysis was an overall relative change in benefits of 5.4 percent from the model baseline as the result of the sixteen Florida 2015 SAR investments funded by HSGP.

Conclusions The DOM pilot leveraged best practices in the application of SME opinion, augmented that opinion with additional simulation, and established that the approach failed to provide confidence sufficient either to meet the requirements of FEMA or to become an HSSEDI recommendation for additional pursuit. In the absence of cause and effect relationship measures, outcomes from previous investments, and inconsistent and non-standardized investment descriptions, this approach used manual sorting to align projects to targets, simulation to assess the contribution of SAR projects to targets, and emergency response SMEs to assess the degree

viii

of specific target contributions to benefits. The use of these assumptions, assertions, and SME inputs significantly reduces the level of confidence in the results.

Recommendations HSSEDI recommends that FEMA consider the complications and issues revealed by the pilot to resolve the ambiguity and cross-correlation that exists in the targets, core capabilities, and investment descriptions, along with the degree to which investments contribute to the achievement of targets. Through the attempt to construct the relationships between investments, capabilities, targets, and observations of performance, the DOM pilot revealed that substantial work is needed to fuse the efforts into a cohesive and unambiguous network or system. The cross-correlation between capabilities and targets that occurs as the result of the absence of a standard lexicon for the categories will continue to obscure any attempt to apply a disciplined approach to quantifying these relationships. FEMA should consider an effort to standardize these relationships for clarity in the development of projects/investments and to provide clear direction for estimating and measuring performance. The SME tables and input from this effort may serve as a starting point to resolve ambiguity; however, there is the potential that both targets and capabilities may need to be re-developed to remove the cross-correlation. On the basis of this pilot, HSSEDI recommends that FEMA address data gaps, identified in Appendix A.6, and implement the data recommendations included in Appendix A.6.1, in order to improve FEMA’s ability to assess the value of the grant program in the future. The value tree weights for assessing the benefit categories in terms of overall benefit are subjective and should be reassessed by FEMA and its stakeholders if this methodology is implemented and used to support funding decisions.

Revealed Preference Analysis

Objective The purpose of this study was to evaluate whether one could use the actual allocation of SHSP funds to infer the magnitude of the perceived value of applications to the SHSP program and the extent to which that value exceeds the size of the grant requests. Virginia’s allocation of SHSP funds was used as a pilot study because Virginia’s formal vetting process generates a rich and convenient data set that enabled HSSEDI to conduct a feasibility test. The total monetized value of everything someone is willing to give up is called “willingness to pay” (WTP). This analysis measures the state’s implicit WTP by analyzing the statistical tradeoff between project costs and other project characteristics. In short, the value of the project to senior decision makers (SDMs) is the maximum amount of funds that they would be willing to commit to that project.

Approach This pilot attempted to value SHSP projects in Virginia by using logistic regression to estimate the emergency management SDMs’ demand for these projects. WTP can be derived from the

ix

demand relationships by a simple formula that quantifies the area under the demand curve. The technique assumes that the SDMs utility depend on the project scores, the requested funds, and other factors that cannot be observed in the data. The technique was applied for every year in which data were obtained from the Virginia Department of Emergency Management (VDEM)—2012 through 2017.

Results The results confirmed prior expectations that the chances of having a proposal approved increases with the final score of reviewers and decreases with the requested funds, the exact magnitudes of each effect change over time. All of this is expected given VDEM’s formal decision-making process. For example, when SMEs evaluated and graded proposals in 2017, every point that they increased a proposals final score raised the probability that the proposal would be selected by an average of 0.84 percent. Similarly, every $1,000 increase in each proposal’s funding request in 2017 lowered the probability that the proposal would be selected by 0.65 percent, on average. These relationships were highly statistically significant in 2017, similar to the models estimated for previous years. The demand relationship inferred from the 2017 SHSP allocation data suggests SDMs implicitly valued each increase in a proposal’s scored point by a dollar value of $1,328 in 2017. However, the 95 percent confidence interval of this ratio falls between $807 per point and $1,723 per point. These results are close in magnitude to the 2016 estimates as well. Finally, the demand curves were used to calculate the WTP for every SHSP proposal that was chosen by SDMs. In 2017, the total perceived value for all proposals that were awarded in this competitive process was $4.7 million, while the total funds requested was $2.5 million. Therefore, in 2017, SDMs were estimated to have perceived approximately 90 percent more value in those grants that were competitively awarded relative to the size of the requested grant funds (i.e., a benefit-cost ratio of 1.9).

Conclusions The revealed preference pilot demonstrated the feasibility of inferring how much SDMs value preparedness and prevention grants. Conclusively, Virginia’s SDMs value the grants selected by nearly twice as much as those grants cost. However, this conclusion only applies to grants that were competitively awarded in VDEM’s formal process of allocating SHSP funds. A successful extension of this pilot would likely only be applicable to grants awarded in other states and urban areas that also undergo a formal, competitive vetting process. It should be emphasized that the valuation estimated in this pilot pertains to decision makers, not the public. Because SHSP funds are ultimately borne by the public through higher taxes, an ideal cost benefit would estimate the value to the public. This has proven to be elusive for HSGP projects due to the high degree of heterogeneity amongst the projects and limited information on the impact of each project. Extending the pilot should only be made if knowing perceived SDM value is a useful criterion to FEMA and OMB for judging program effectiveness.

x

Recommendations Replicating the Revealed Preference analysis to other State, Local, Tribal, and Territorial (SLTT) governments depends on allocation processes that mirror Virginia’s. Specifically, they must have an evaluation process that produces scored output(s) for each project, and the program must document which projects were ultimately selected for funding and which were not. The (likely) variation in scoring methods and data collection across SLTT produces the caveat that RPA results may not be comparable among jurisdictions. MITRE recommends canvasing the SLTT to determine how much of the HSGP funds are allocated via such formal processes. Additionally, further statistical and analytical work beyond the RPA would be required to ascertain which comparisons and conclusions could be made at the national level.

Breakeven Analysis

Objective The BA treats projects as contributions to a national capability, as opposed to the capability of a specific state, locality, or region. The objective is to construct a national-level estimate of the losses from terrorism activities (fatalities, injuries, and property damage,) in the United States. The focus of the BA is to explore how effectively the investments in the HSGP prevention portfolio may reduce those losses. This document provides detailed information about the inputs, processes, and outputs of that analysis. The BA is an exploratory pilot/feasibility study and does not produce findings that prove causality. The results should be interpreted with caution given the nature of the methodology.

Approach HSSEDI developed a model to explore the correlation between core capabilities and project outcomes. SMEs translated project outcomes within a sample of core capabilities to loss reductions. The loss reductions were translated into Net Benefit Cost Ratio (NBCR) ranges. The model produced a NBCR range for each core capability.

Conclusions The BA demonstrated that a quantitative investment benefit measure could be developed for the prevention portfolio within the HSGP. The BA approach supplemented existing FEMA Biannual Strategy Implementation Report (BSIR)/IJ project data with data from the Global Terrorism Database, the expertise of terrorism prevention SMEs, and an analytical model. It appears that other portfolios within the HSGP, like the protection portfolio, would be conducive to analysis using the BA. It is also possible that the expertise of SMEs could be replaced in the future by a combination of improved project data and simulation models, but that remains to be assessed.

Recommendations HSSEDI has the following recommendations:

xi

• Identify the other HSGP portfolios that could be assessed with the BA method. There are five mission area portfolios in all: Prevention, Protection, Mitigation, Response, and Recovery. Of those, the first three could be assessed with the BA methodology. In addition to their primary impact, Protection and Mitigation have a terrorism deterrence effect associated with them. Projects in those portfolios could be assessed for both reduction in terrorist attack frequency (due to deterrence) and reduction in impacts. The Response and Recovery portfolios do not have a deterrence aspect to them; hence, assessing them with the BA methodology would require revisions to the model and data specific to response and recovery. Significant HSGP budget coverage by the Protection and Prevention portfolios is shown, for example, by the fact that Protection and Prevention combined account for 60 percent of HSGP budget on average for the years 2010 through 2015.

• In the near term, help the SMEs better understand how to express uncertainty ranges.

• In the longer term, assess whether SMEs can ultimately be replaced by improved data and simulation models in future applications of the BA:

o Improved data could include, for example, the mapping of each project to its THIRA target. It could also include the amount of quantitative capability improvement in whatever terrorism prevention or protection capability the project provides.

o Determine how available and scalable simulation models are for the types of HSGP projects that represent core capabilities.

• Determine if those models simulate reductions in probability of attack and likelihood of impacts resulting from FEMA HSGP projects.

xii

Table of Contents 1 Introduction ......................................................................................................................................... 17

1.1 Requirements ................................................................................................................................ 181.2 Constraints and Assumptions ..................................................................................................... 19

1.2.1 Constraints ............................................................................................................................. 191.2.2 Assumptions .......................................................................................................................... 19

2 Detailed Operations Model ................................................................................................................. 222.1 Approach ....................................................................................................................................... 222.2 Assumptions ................................................................................................................................. 252.3 Model Framework – Multi-Attribute Utility Theory ..................................................................... 25

2.3.1 Value Tree Structure .............................................................................................................. 272.3.2 Assessing the Impact of THIRA Targets on Achieving Major Benefits ............................. 282.3.3 Assessing Impact of Benefit Categories to Overall Benefit ............................................... 312.3.4 Assessing Impact of Investments to Achievement of THIRA Targets ............................... 31

2.4 Simulation ..................................................................................................................................... 312.5 DOM Output .................................................................................................................................. 322.6 DOM Pilot ...................................................................................................................................... 32

2.6.1 Review of Investments .......................................................................................................... 322.6.2 Jurisdiction Alignment to Investments ................................................................................ 322.6.3 Simulation ............................................................................................................................... 33

2.7 Simulation Assumed Values ........................................................................................................ 342.8 Pilot Results .................................................................................................................................. 352.9 Scaling the Detailed Operations Model ...................................................................................... 372.10 Conclusions and Recommendations .......................................................................................... 39

2.10.1 Conclusions ........................................................................................................................... 392.10.2 Recommendations ................................................................................................................. 39

3 Revealed Preference Analysis of Virginia SHSP Projects ............................................................... 413.1 Overview ........................................................................................................................................ 413.2 Description of Data ....................................................................................................................... 42

3.2.1 Virginia’s Multi-Objective Decision Model Design .............................................................. 433.2.2 Final Sample Characteristics ................................................................................................ 48

3.3 Methodology ................................................................................................................................. 53

xiii

3.4 Results........................................................................................................................................... 563.5 Discussion .................................................................................................................................... 58

3.5.1 Limitations .............................................................................................................................. 593.5.2 Recommendations ................................................................................................................. 60

4 Breakeven Analysis ............................................................................................................................ 624.1 Overview ........................................................................................................................................ 624.2 Approach ....................................................................................................................................... 634.3 BA Process Flow .......................................................................................................................... 65

4.3.1 Net Benefit Cost Ratio Model ................................................................................................ 664.3.1.1 Computing Net Benefit Cost Ratio ............................................................................. 66

4.4 Data Sources ................................................................................................................................. 674.5 BA Results .................................................................................................................................... 70

4.5.1 SME Elicitation ....................................................................................................................... 704.6 SME Calibration Training and Elicitation .................................................................................... 71

4.6.1 SME Assessment of the Project Investments ...................................................................... 724.6.2 Confidence in SME Results ................................................................................................... 734.6.3 Graphing and Tabulating the Net Benefit Cost Ratios ........................................................ 73

4.7 Findings ......................................................................................................................................... 754.8 Recommendations ....................................................................................................................... 76

5 Next Steps ........................................................................................................................................... 785.1 Transition to Production Assessments ...................................................................................... 785.2 Recommendation ......................................................................................................................... 79

Appendix A: DOM ...................................................................................................................................... 80A.1 Definition of Benefit Categories .................................................................................................. 80A.2 SAR Subject Matter Experts ........................................................................................................ 81A.3 Calculation of Investment Benefit ............................................................................................... 82A.4 Simulation Inputs and Sources ................................................................................................... 84A.5 Data Sources ................................................................................................................................. 85A.6 Data Findings ................................................................................................................................ 87

A.6.1 Data Improvement Recommendations ................................................................................. 87Appendix B: Breakeven Analysis ............................................................................................................. 90

B.1 New Algorithm for Combing Core Capabilities at the Prevention Portfolio Level .................. 90B.2 Sample Detailed Calculation of a Prevention Portfolio Net Benefit Cost Ratio ....................... 92

xiv

B.3 BA Subject Matter Experts ........................................................................................................... 94B.4 Project Samples ............................................................................................................................ 95B.5 Expert Judgement Elicitation and Script .................................................................................. 100B.6 Percentage of Dollars Represented by the Sample of Prevention Portfolio Projects .......... 102B.7 SME Responses .......................................................................................................................... 104B.8 Project Selection Criteria and How They Were Fulfilled ......................................................... 111B.9 Net Benefit Cost Ratio for Core Capability ............................................................................... 119

List of Acronyms ..................................................................................................................................... 122List of References.................................................................................................................................... 125

Table of Figures Figure 1. Alignment of Investments to Targets to Benefits .................................................................... viFigure 2. Relationship Between Grants, Investments, and Projects ..................................................... 22Figure 3. Alignment of Investments to Targets to Benefits ................................................................... 23Figure 4. Process for Estimating Benefit of HSGP Projects .................................................................. 24Figure 5. Elements of the FEMA Value Tree ............................................................................................ 27Figure 6. Systems Dynamics View ........................................................................................................... 34Figure 7. Demand Curve and Willingness to Pay .................................................................................... 42Figure 8. Distribution of Final Proposal Scores, by Year and Approval Status ................................... 51Figure 9. Distribution of Proposal Requested Funds, by Year and Approval Status ........................... 52Figure 10. Heuristic SDM Demand for 10 Proposals that All Have Matching Scores and Costs ........ 54Figure 11. Heuristic Demand Curves for Two Proposals Wtih Different Reviewer Scores ................. 55Figure 12. Distribution of Individual Proposal Net Benefits ($) by Year and Approval Status ............ 58Figure 13. BA Process Flow ..................................................................................................................... 65Figure 14. Sample SME Elicitation Page .................................................................................................. 71Figure 15. Mapping of Overall Prevention Portfolio Net Benefit Cost Ratio Range ............................. 74Figure 16. Lower Bound on the Probability Reduction .......................................................................... 90Figure 17. Upper Bound on the Probability Reduction .......................................................................... 91Figure 18. Mapping of Intelligence and Information Sharing Net Benefit Cost Ratio Range ............ 119Figure 19. Mapping of Interdiction and Disruption Net Benefit Cost Ratio Range............................. 119Figure 20. Mapping of Screening, Search, and Detection Net Benefit Cost Ratio Range ................. 120Figure 21. Mapping of Public Information and Warning Net Benefit Cost Ratio Range .................... 120Figure 22. Mapping of Forensics and Attribution Net Benefit Cost Ratio Range ............................... 121

xv

Table of Tables Table 1. SME Scores for SAR Target ........................................................................................................ viTable 2. Major Benefit Category Contribution to Overall Investment Benefit ...................................... viiTable 3. Assessment Scale ....................................................................................................................... 29Table 4. Mapping Between Targets and Categories ............................................................................... 29Table 5. Major Benefit Category Contribution to Overall Investment Benefit ...................................... 31Table 6. Example of Results from a Simulation Run .............................................................................. 37Table 7. Questions and Question Weights Used by VDEM in the SHSP Multi-Objective Decision Model .......................................................................................................................................................... 45Table 8. Count of Variables and Eigenvalues > 1 in Principal Components Analysis Performed on Reviewer Responses, by Year .................................................................................................................. 48Table 9. Counts and Mean Characteristics of Sample ............................................................................ 49Table 10. Total Amount of Requested and Awarded Funding, by Competitive Process ..................... 50Table 11. Coefficient Estimates from Logistic Regressions of Proposal Approval Status, by Year .. 56Table 12. WTP per Point, by Year ............................................................................................................. 56Table 13. Aggregate Proposal Benefits and Costs by Year and Approval Status ............................... 57Table 14. Prevention Core Capabilities and Sample Projects ................................................................ 68Table 15. GTD Provides a Basis to Compute the Uncertainty of Terrorism Losses ............................ 69Table 16. Total Prevention Portfolio, Combination of the Five Core Capabilities ................................ 73Table 17. Summary of Net Benefit Cost Ratio Ranges ........................................................................... 75Table 18. BA Results ................................................................................................................................. 75Table 19. Comparison of Benefit Categories ........................................................................................... 80Table 20. SME Qualifications .................................................................................................................... 81Table 21. Calculation of Individual Investment Benefit .......................................................................... 82Table 22. Simulation Inputs, Sources, and Descriptions ....................................................................... 84Table 23. Data Sources Reviewed ............................................................................................................ 85Table 24. Utility Score Descriptions ......................................................................................................... 87Table 25. BA SME Panels .......................................................................................................................... 94Table 26. FEMA Prevention Portfolio Core Capability – Project Samples ............................................ 95Table 27. Interdiction and Disruption ....................................................................................................... 96Table 28. Public Information and Warning .............................................................................................. 97Table 29. Screening, Search, and Detection............................................................................................ 99Table 30. Forensics and Attribution ......................................................................................................... 99Table 31. Prevention Portfolio Sample Projects ................................................................................... 102Table 32. SME Response: Intelligence and Information Sharing ........................................................ 104

xvi

Table 33. SME Response: Interdiction and Disruption ......................................................................... 105Table 34. SME Response: Public Information and Warning ................................................................ 107Table 35. SME Response: Screening, Search, and Detection .............................................................. 108Table 36. SME Response: Forensics and Attribution ........................................................................... 109Table 37. “Critical Tasks” for Each Core Capability ............................................................................. 111Table 38. Project Samples Annotated with Critical Task Numbers ..................................................... 114

17

1 Introduction Each year the Department of Homeland Security (DHS) Federal Emergency Management Agency (FEMA) distributes approximately $1 billion in grants to State, Local, Tribal, and Territorial (SLTT) governments to improve preparedness against natural, technological, and man-made hazards.6 The SLTT stakeholders expend these funds on a variety of projects to build and sustain disaster preparedness capacity across the 32 core capabilities within five mission areas identified as critical to achieve the National Preparedness Goal.7

FEMA’s National Preparedness Assessment Division (NPAD) is charged with assessing and communicating the impacts of these investments, activities, and accomplishments as they relate to national preparedness.8 Their responsibilities include assessing the effectiveness of FEMA programs, such as the Homeland Security Grant Program (HSGP), which comprises the State Homeland Security Program (SHSP), Operation Stonegarden (OPSG), and the Urban Area Security Initiative (UASI) to improve national preparedness. NPAD engaged the Homeland Security Systems Engineering and Development Institute (HSSEDI) to support their efforts to understand the return on investment (ROI) provided by the FEMA emergency preparedness grants. HSSEDI is conducting this work in two phases. In Phase I, HSSEDI determined that the existing FEMA data were insufficient to generate a monetized ROI and recommended pursuing three pilots to explore ways to determine the investment benefits of the HSGP. This document is Part 1 of Phase II. It provides the results of three pilots—the Detailed Operations Model (DOM), the Revealed Preference Analysis (RPA), and the Breakeven Analysis (BA).9

The DOM implemented a value tree based on multi-attribute utility theory (MAUT). HSSEDI used a constructive approach to link investments and, as available, the projects within them, to overall benefit. This approach developed an assessment of potential utility, then identified the contribution from that utility, then performed a Threat and Hazard Identification and Risk Assessment (THIRA), and finally produced a derived collection of benefits. The output of the DOM was expressed as a change in benefit where benefit was defined as a reduced risk of negative consequences. The HSSEDI team leveraged a literature search1011 to identify benefits and found six commonly used categories—lives lost, injuries/illnesses, damage, business losses, time spent without necessities, and impact on other quality of life (QOL) issues. These elements

6 Totals to $1.04 billion in FY17. https://www.fema.gov/fiscal-year-2017-homeland-security-grant-program 7 https://www.fema.gov/national-preparedness-goal 8 https://www.fema.gov/national-preparedness-directorate 9 Phase 1 FEMA NPAD Homeland Security Grant Program Return on Investment Methodology, 23 March 2018. 10 Senn, M., A Comprehensive Risk Index for the United States, 2014. http://artsandsciences.sc.edu/geog/hvri/sites/sc.edu.geog.hvri/files/attachments/Senn%20Dissertation%20Final.pdf 11 Hugengusch, D. and Neumann T., Cost Benefit Analysis of Disaster Risk Reduction, Aktion Deutschland Hilft e.V., October 2016. https://www.google.com/search?ei=ndjHW9XKOsiG5wKWuKP4CA&q=Cost+Benefit+Analysis+of+Disaster+Risk+Reduction%2C+Aktion+Deutschland+Hilft&oq=Cost+Benefit+Analysis+of+Disaster+Risk+Reduction%2C+Aktion+Deutschland+Hilft&gs_l=psy-ab.3...212352.214641..215320...0.0..0.446.897.0j1j1j0j1......0....1j2..gws-wiz.......0i71.9-qmR_B9dgw

https://www.fema.gov/fiscal-year-2017-homeland-security-grant-program

https://www.fema.gov/national-preparedness-goal

https://www.fema.gov/national-preparedness-directorate

http://artsandsciences.sc.edu/geog/hvri/sites/sc.edu.geog.hvri/files/attachments/Senn%20Dissertation%20Final.pdf

https://www.google.com/search?ei=ndjHW9XKOsiG5wKWuKP4CA&q=Cost+Benefit+Analysis+of+Disaster+Risk+Reduction%2C+Aktion+Deutschland+Hilft&oq=Cost+Benefit+Analysis+of+Disaster+Risk+Reduction%2C+Aktion+Deutschland+Hilft&gs_l=psy-ab.3...212352.214641..215320...0.0..0.446.897.0j1j1j0j1......0....1j2..gws-wiz.......0i71.9-qmR_B9dgw



18

were compared to the THIRA and to FEMA’s core capabilities to capture the concept of benefit for the pilot. To develop the link between investment or project and utility, systems dynamics models were developed to provide understanding of the relationship between THIRA-associated activities, utility, and the translation of the change in utility into a predicted change in benefit. The DOM pilot leveraged best practices in the application of subject matter expert (SME) opinion, augmented that opinion with additional simulation, and established that the approach failed to provide confidence sufficient either to meet the requirements of FEMA or to become an HSSEDI recommendation for additional pursuit. The RPA of Virginia SHSP projects estimates the value of project proposals submitted for funding by the SHSP in Virginia by measuring the value that senior decision makers (SDMs) in emergency management implicitly place on these projects. For the BA, HSSEDI developed a model to explore the correlation between core capabilities and project outcomes. SMEs translated project outcomes within a sample of core capabilities to loss reductions. The loss reductions were translated into Net Benefit Cost Ratio (NBCR) ranges. The model produced an NBCR range for each core capability. This document provides detailed information about the inputs, processes, and outputs of the analysis.

1.1 Requirements The pilots were to explore whether or not an assessment of the contribution of project investments toward preparedness targets could be conducted without the data required to do an ROI. The pilots were to satisfy the following:

• The approach must be compliant with the Office of Management and Budget (OMB) Circular A-94, which states that benefit-cost analysis (BCA) is the recommended technique for a formal economic analysis of government programs or projects. In cases where the monetary values of some benefits or costs cannot be determined, a comprehensive enumeration of the different types of benefits and costs, monetized or not, can be helpful in identifying the full range of program effects.12

• The assumed values for inputs to the model must be easy to change at all levels.

• The model must infer, assume, assert, simulate, or generate outcomes of investments from project descriptions that exist in current investment documentation (i.e. Biannual Strategy Implementation Reports [BSIRs]).

• The grants data did not include cause-effect relationship measures or outcomes from previous investments, so assumptions are required for both.

12 OMB Circular A-94: Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs.

19

1.2 Constraints and Assumptions Data availability impacted the design of the pilot studies. Assumptions had to be made to account for missing data and all pilots were constrained by the missing data. In many cases, the assumptions and constraints that affected the pilots were the same. In situations where assumptions and constraints vary, they are called out in the specific pilot discussions.

1.2.1 Constraints The major constraints applicable to development of the DOM were:

• Data gaps in four major categories: o No preparedness capability baseline o Inconsistent and non-standardized investment descriptions o No defined benefit measures o No description of the link(s) between investment and benefit; the information for

each grant project was limited to a short text description, which challenged the SMEs to confidently assess benefit(s)

• SME elicitation was limited to a single session due to contract deadlines The major constraints applicable to development of the RPA were:

• The RPA presented here does not measure the value of the SHSP in Virginia to the public owing to the cost and time required to collect these data (e.g., through a survey). Instead, the RPA was performed on proposal- scoring data by SDMs during Virginia’s allocation process.

• Analysis performed using Virginia data is not necessarily comparable to analyses in other states, owing to the specifics of Virginia's emergency management needs and the nuances of the scoring methodology.

The major constraints applicable to development of the BA were:

• Because of time constraints, the BA was limited to the Prevention mission area. • The BA was limited to preventing terrorism-related disasters due to the lack of national-

level datasets to assess impact on other types of disasters. • The Global Terrorism Database (GTD) has no historic data for the threat profiles and

costs of terrorists’ use of weapons of mass destruction that have not actually been used in the United States (e.g., an improvised nuclear device). Hence, the benefits of projects acting to prevent such threats might be underestimated.

1.2.2 Assumptions The following assumptions were necessary to address the constraints described in Section 1.2.1 above. The primary assumptions made during development of the DOM were:

20

• Investments and their projects include descriptions of tangible results that can be aligned to set of defined benefits.

• THIRA targets link to the major benefit categories applied to all HSGP grants.

• Data collection methods vary by state; the most meaningful investment and reporting data are available at the state level.

• Investment benefits are gathered and measured on an annual basis. The major assumptions applicable to the development of the RPA were:

• Logistic regression was used to estimate the implicit demand curve of SDMs for projects. The technique assumes that the SDMs have some unknown utility dependent on project scores and reflecting requested funds.

• The SDMs may also use other unobserved information when deciding whether they should fund a project. That unobserved component is assumed to be random and independent of other factors (e.g., score and cost).

• Some of the statistical assumptions required by the logistic regression model are unlikely to hold. HSSEDI found that errors are correlated across observations: the selection of one award reduces the chances of other awards being selected.

• The willingness to pay (WTP) formula values every grant using the estimated marginal rate of substitution between score and cost. If SDMs have diminishing marginal utility with respect to score, for example, it would be inappropriate to apply a marginal concept in the valuation of an entire grant. Because the benefit/cost ratio is emphasized during the allocation process, the assumption of constant marginal value was determined to be appropriate for Virginia.

The major assumptions made during the development of the BA are: • HSSEDI selected a sample of grant projects from a large set of projects for each of the

chosen core capabilities. The sample of projects chosen represent national-level core capabilities that contain hundreds or thousands of projects.

• Local and regional projects can represent national-level capabilities; individual projects can be considered representative of similar local-level projects being funded in many states across the nation.

• Projects only provide benefit to a single core capability and single portfolio.

• Project impacts are independent of each other—it is assumed the reduction in likelihood and impact probabilities of one project is not dependent on what other projects have been implemented)—while core capabilities are not independent of each other. Their overlap must be accounted for in the methodology.

• Project and core capability benefits are measured in terms of reduction in terrorist attack likelihoods and impacts. Additional potential benefits from the same set of projects for non-man-made disasters are not considered.

21

• Project and core capability benefits are measured in terms of reduction in terrorist attack likelihood and impacts. Potential benefits for non-man-made disasters are not considered. The rationale was that only the GTD was used to obtain losses; hence, no data were available to assess losses from non-man-made disasters.

22

2 Detailed Operations Model 2.1 Approach The DOM implemented a value tree based on MAUT. The HSSEDI team’s approach was to link investments and, as available, the projects within them, to overall benefit. This approach built from an assessment of potential utility, to the contribution from that utility, through the THIRA, and to a derived collection of benefits. The output of the DOM was expressed as a change in benefit where benefit was defined as a reduced risk of negative consequences. The HSSEDI team leveraged a literature search to identify benefits and found six commonly used categories that aligned with elements of the THIRA and core capabilities—lives lost, injuries/illnesses, damage, business losses, time spent without necessities, and impact on other QOL issues.1314

To develop the link between the investment or project and utility, HSSEDI developed system dynamics models to provide understanding of the relationship between THIRA-associated activities, utility, and the translation of the change in utility into a predicted change in benefit. SMEs assessed the contributions of THIRA targets toward benefit categories and input them into the value tree. Figure 2 shows the relationship between grants, investments, and projects. Each grant is composed of 1 to N number of investments and each investment is composed of 1 to M projects. This relationship provided the challenge to identify, estimate, allocate, and combine effects across the investment and project-level documentation.

Grants

Investments (1, 2, …, N)

Projects (1, 2, …, M)

Figure 2. Relationship Between Grants, Investments, and Projects The DOM asserts that project contributions from an investment can be aligned to THIRA targets and the contribution of THIRA targets can be aligned to benefit categories to develop an overall investment benefit score. Figure 3 depicts this alignment.

13 Senn, M., A Comprehensive Risk Index for the United States, 2014. http://artsandsciences.sc.edu/geog/hvri/sites/sc.edu.geog.hvri/files/attachments/Senn%20Dissertation%20Final.pdf 14 Hugengusch, D. and Neumann, T., Cost Benefit Analysis of Disaster Risk Reduction, Aktion Deutschland Hilft e.V., October 2016. https://www.google.com/search?ei=ndjHW9XKOsiG5wKWuKP4CA&q=Cost+Benefit+Analysis+of+Disaster+Risk+Reduction%2C+Aktion+Deutschland+Hilft&oq=Cost+Benefit+Analysis+of+Disaster+Risk+Reduction%2C+Aktion+Deutschland+Hilft&gs_l=psy-ab.3...212352.214641..215320...0.0..0.446.897.0j1j1j0j1......0....1j2..gws-wiz.......0i71.9-qmR_B9dgw

http://artsandsciences.sc.edu/geog/hvri/sites/sc.edu.geog.hvri/files/attachments/Senn%20Dissertation%20Final.pdf




23

Overall benefit

Major benefit (risk reduction) categories

THIRA targets

Investments

Figure 3. Alignment of Investments to Targets to Benefits

The analysis of the DOM approach was based on an assessment of the potential to develop estimates, allocations, and combination logic. Where Figure 2 shows the hierarchical relationship of grants, investments, and project documentation, Figure 3 provides the connection from investment or project to estimates of effect aligned to THIRA and then to benefit. HSSEDI elected to treat the alignment, allocation, and estimation elements as independent portions of the model. The two major components of the DOM model are:

1. Value Tree – Leveraging multi-attribute utility theory, HSSEDI established a hierarchy of utility or value tree: projects contribute to achieving THIRA targets, and meeting targets contributes to benefits. The value tree structure is described in detail in Section 2.2.1.

2. System Dynamics Simulation – System dynamics simulation estimated the contribution of projects toward THIRA targets. The simulation component, described in detail in Section 2.5.3, provided the estimate that was an input to the value tree.

The identification and allocation of projects were based on actual data. Running this model with THIRA targets remains an area for additional exploration. Figure 4 illustrates the two key components and the process for estimating the benefit of HSGP projects using these components and SME inputs. The funneling and grouping at the top represents the allocation effort; once projects are aligned to THIRA targets, the system dynamics simulations estimated the effects of intended changes to benefit categories, and that effect was used as a utility score in the value tree to drive a cross-target change in overall benefit.

24

Figure 4. Process for Estimating Benefit of HSGP Projects

The process includes the following steps: 1. Manually review investments for applicability. Extract investments related to core

capability (e.g., mass search and rescue) from BSIR. 2. Map investments to THIRA targets. 3. Group investments into major benefit categories. Assess the percent change or

contribution of investments toward THIRA targets by simulating investment effects on the rate of rescue by location.

4. Input investments’ contribution toward THIRA target simulation results into value tree structure.

25

5. SMEs assess the contributions of THIRA targets toward benefit categories and input the evaluation of the contributions into value tree structure.

6. Assess the impact of benefit categories to overall benefit. 7. Roll up benefits/risk reduction for all search and rescue (SAR) HSGP investments using

value tree structure.

2.2 Assumptions The primary assumptions made in the development of the DOM are:

• Investments and their projects all produce tangible results in terms of a set of benefits.

• The link between the THIRA targets and the major benefit categories applies to all HSGP grants.

• The link between investments to THIRA targets needs to be assessed by each state because data collection methods vary by state; the most meaningful investment and reporting data are available at the state level.

• Investment benefit is gathered and measured on an annual basis.

2.3 Model Framework – Multi-Attribute Utility Theory In pursuing the DOM approach, HSSEDI carried out a value-based analysis following the principles of MAUT. This approach forces identification of key objectives and capability gaps and provides a framework for linking investment options to improved capability. Shreve and Kelman15 conclude, based on their review of multiple cost-benefit analyses of disaster risk reduction, that this type of approach “may be more efficient at highlighting social and environmental vulnerabilities and thus benefits than CBA [cost benefit analysis] alone.” The benefit of the investment options can be assessed using the concept of a value tree. The value tree identifies the high-level mission and decomposes it hierarchically (e.g., into goals, objectives, actions, and tasks) down to the level at which the effect of the investment options can be directly evaluated. The tree thus provides a step-by-step framework for linking investment options to the high-level mission—overall improved preparedness capability. The impact of each investment option is assessed at the lowest level of the tree in terms of a utility score, a dimensionless value on a 0-100 scale. For the scale and scope of the investments considered by the DOM, hundreds of assessments are required, therefore HSSEDI chose to use the system dynamics model to make these assessments. At every other level of the value tree, “rollup rules” are defined to specify the relative contribution of each “child” node to achieving the capability represented by its “parent” node.16

15 Shreve, C.M., and Kelman, I., “Does mitigation save? Reviewing cost-benefit analyses of disaster risk reduction,” International Journal of Disaster Risk Reduction, 10 (2014), pp. 213–235. 16 A weighted average is commonly used as a rollup function, but many other options are possible.

26

The rollup rule describes how the benefit scores of the “child” nodes combine to produce the benefit score at the “parent” node. The number of possible rollup rules is limitless; any algebraic expression can be used, and this expression can be simple or complicated, depending on how simple or complicated the interactions and interdependencies among the child node benefits are. The nature of these interactions and dependencies is therefore a driver for defining the rollup rule. Another determining factor is how much information about those relationships, interactions, and interdependencies is available to the analyst. For the DOM pilot effort, no information was available a priori about these interactions and interdependencies. Therefore, HSSEDI decided to use one of the most commonly employed rollup rules: a weighted average. For this rule, the only information required is an assessment of the relative importance of the child node benefits to determine the weights. The team felt that it would be possible to make credible first-cut assessments of the weights based on expert judgment. Had the pilot effort continued into further development and implementation, HSSEDI would have worked with FEMA NPAD to further evaluate and refine the choice of rollup rules and to better define the rules’ parameters (for example, the weights in a weighted average). The complete set of rules, at all levels of the tree, allows the benefit at the top of the tree to be calculated from the scores of the lowest nodes. The overall benefit to the mission of each individual investment or project can thus be calculated. Furthermore, the extent to which the high-level mission is being achieved by each state based on all its individual projects can be computed. The process of developing the tree structure is an essential element of the analysis and should be performed by FEMA; it demands answers to questions like, “What are FEMA’s overall goals and objectives for the grant program?” and “What does HSGP ‘success’ look like?” The value tree used in the DOM is shown in Figure 5. The overall preparedness benefit of an investment or project (the green box at the top) is assessed in terms of the level of benefit achieved in each of six major benefit categories (the blue boxes). These categories represent six different aspects of reducing the negative consequences of a disaster. A weighted average rollup function reflecting the relative importance of each benefit category prescribes how the scores of all the benefit categories are combined to produce the overall “preparedness” score. HSSEDI found no reference or data elements in the information reviewed that could provide the weights for these elements. As a result, an elicitation process was developed using HSSEDI SMEs to provide the weights. The benefit category scores, in turn, are derived from the target scores at the next lower level in the tree (the orange boxes) using another weighted average rollup function. The weights used in this rollup function were also developed by HSSEDI SMEs.

27

Overall Benefit

Reduced risk of lives lost

Applicable THIRA target or targets

Reduced risk of injuries/illnesses


Reduced risk of property damage


Reduced risk of business losses


Reduced risk of time without necessities


Reduced risk of other quality of

life issues


Figure 5. Elements of the FEMA Value Tree

Each investment or project (not shown in the figure) was assessed in terms of how well it meets the relevant target, in this case the SAR target. A full model would include all the THIRA targets.

2.3.1 Value Tree Structure HSSEDI explored numerous ways to structure the value tree for analyzing the investments funded by the HSGP grants. One option was a tree structure based on the National Preparedness Goal hierarchy.17 The National Preparedness Goal is “A secure and resilient nation with the capabilities required across the whole community to prevent, protect against, mitigate, respond to, and recover from the threats and hazards that pose the greatest risk.” It includes five mission areas—Prevention, Protection, Mitigation, Response, and Recovery—each of which is supported by one or more of 32 core capabilities. The document also identifies preliminary targets associated with each core capability. However, this tree structure was determined to be at too high a level to be able to effectively make the link to individual investments. Another option was the value tree hierarchy used by the Virginia Department of Emergency Management (VDEM) for prioritizing investments for competitive award.1819 However, the metrics used in this structure were largely programmatic rather than being focused on benefit/performance. HSSEDI also reviewed the grant effectiveness case studies conducted by FEMA NPAD for Ohio, Montana, Maine, and the District of Columbia. Based on this research and analysis of possible methods, HSSEDI selected a cause-and-effect value tree approach. This tree structure sought to demonstrate the relationship between emergency preparedness grants (costs) and reductions in the risk of loss of life, injuries/illnesses,

17 National Preparedness Goal, Second Edition, September 2015 18 Governor McAuliffe Announces $5.7 Million in Homeland Security Grant Awards - Virginia Department of Emergency Management 19 Ezell, B. Lawsure, K. and Flanagan, D., “Risk and Decision Analytic Support to the Commonwealth of Virginia State Homeland Security Program,” Final Report, December 31, 2015 updated: February 2, 2016, Virginia Modeling, Analysis and Simulation Center.

http://www.vaemergency.gov/governor-mcauliffe-announces-5-7-million-homeland-security-grant-awards/


28

property damage, and economic losses. The top of the tree represents the overall benefit of the investment. The next layer of the tree breaks the overall benefit down into six major benefit categories intended to capture all aspects of the risk that should be reduced. The categories are designed to be independent:

• Reduced risk of lives lost

• Reduced risk of injuries/illnesses

• Reduced risk of property damage

• Reduced risk of business losses

• Reduced risk of time spent without necessities

• Reduced risk of impact on other QOL indicators These categories were defined by HSSEDI based on those defined in Benefit-Cost Analysis of FEMA Hazard Mitigation Grants by Rose, et al.20 and Cost-Benefit Analysis of Disaster Risk Reduction by Aktion Deutschland Hilft e.V.21 The table in Appendix A.1 describes the similarities between the categories chosen for the DOM pilot and these references. The next layer of the value tree links the major benefit categories to the 24 targets identified by FEMA for the 2018 THIRA. The THIRA targets were chosen because they arise from an established FEMA framework and can be linked logically to both the major benefit categories and the HSGP investments. At the lowest level of the tree, the impact of the investment options on meeting the THIRA targets is assessed.

2.3.2 Assessing the Impact of THIRA Targets on Achieving Major Benefits For this pilot, several HSSEDI SMEs provided impact assessments of THIRA targets on achieving major benefits.22 Summaries of the 2018 THIRA targets were used for brevity. The targets were mapped to the benefit categories using the scale shown in Table 3. HSSEDI defined a relative scale where medium has a numerical value of 50 and high and low are defined as 30 points above and below medium. The scale was chosen based on experience with previous studies using MAUT. A zero to 100 scale is typically used in assessing utility.23

HSSEDI chose to employ a high-medium-low scale of 80, 50, and 20 because of the highly qualitative nature of the assessments. This scale exhibits both symmetry and balance. “Symmetry” means they contain equal numbers of positive and negative positions whose respective distances apart are bilaterally symmetric about the “neutral” value—in this case, a

20 Rose, A. et al. Benefit-Cost Analysis of FEMA Hazard Mitigation Grants, DOI: 10.1061/(ASCE)1527-6988-(2007)8:4(97) 21 Hugenbusch, D. and Neumann, T., Cost-Benefit Analysis of Disaster Risk Reduction, Aktion Deutschland Hilft e.V., October 2016. 22 The SMEs and their qualifications are listed in Appendix A.2. 23 Pinto, C.A. and Garvey, P.R., Advanced Risk Analysis in Engineering Enterprise Systems, CRC Press, Business & Economics, 19 April 2016.

29

medium contribution of a target toward a major benefit. “Balance” means that the distance between each candidate value is the same.

Table 3. Assessment Scale

Target Impact Description Numerical Value

High Meeting the target makes a significant contribution to achievement of the benefit. 80

Medium Meeting the target makes a moderate contribution to achievement of the benefit. 50

Low Meeting the target makes a minimal contribution to achievement of the benefit. 20

None Meeting the target makes no contribution to achievement of the benefit. 0

Table 4 illustrates the mapping made by the SMEs between the targets and the benefit categories, using the scale in Table 3. The values shown are the average of the inputs from the three SMEs. The values in the table are used as the weights in a weighted average rollup rule to calculate overall benefit. For example, the calculation of the weighted average benefit score for the reduced risk of lives lost due to investments that contribute to the SAR target of “Conduct search and rescue operations” is:

SAR Target score (from simulation is 7.5) * SME assessment for “Reduced risk of lives lost” due to SAR target (from simulation is 80) / Sum of SME assessments for the contribution of all THIRA targets toward “Reduced risk of lives lost” (from simulation is 1320)

7.5 * 80/1320=0.45 The weighted average calculation assumes that, for each investment, an assessment of the change in utility for each of the THIRA targets can be assessed. The collection of THIRA target utility changes would use the table to weight the contribution to the benefit categories and produce estimates of each benefit category.

Table 4. Mapping Between Targets and Categories

THIRA Targets Benefit Categories - Reduced Risk of:

Lives Lost Injuries Property

Damage Business Losses

Time w/o Necessities

Quality of Life

Impact Evacuate people following notice of impending event 80.0 80.0 13.3 40.0 60.0 60.0

Clear roads to enable access for emergency responders 70.0 50.0 40.0 40.0 60.0 60.0

Reopen businesses 6.7 6.7 23.3 80.0 80.0 60.0 Contain and begin cleanup of hazmat release 70.0 70.0 40.0 40.0 23.3 60.0

30

THIRA Targets Benefit Categories - Reduced Risk of:

Lives Lost Injuries Property

Damage Business Losses

Time w/o Necessities

Quality of Life

ImpactDecontaminate individuals exposed to hazmat 70.0 80.0 6.7 23.3 23.3 60.0

Recover, identify, and provide mortuary services for fatalities 0.0 0.0 0.0 13.3 6.7 60.0

Suppress and extinguish structure fires 80.0 80.0 70.0 70.0 60.0 70.0

Restore healthcare and social service functions 80.0 60.0 13.3 23.3 60.0 70.0

Find long-term housing for those displaced 13.3 23.3 6.7 23.3 50.0 70.0

Restore water service 30.0 30.0 23.3 40.0 70.0 43.3Restore wastewater service 23.3 33.3 23.3 23.3 60.0 70.0Restore communication service 50.0 40.0 30.0 40.0 40.0 50.0 Restore power service 50.0 50.0 50.0 60.0 70.0 70.0 Establish and maintain distribution system for necessities 40.0 23.3 13.3 40.0 80.0 60.0

Provide emergency shelter, food, and water 80.0 80.0 33.3 33.3 80.0 60.0

Move people from temporary to permanent housing 13.3 23.3 13.3 13.3 40.0 60.0

Conduct search and rescue operations 80.0 80.0 23.3 23.3 40.0 60.0

Restore natural and cultural resources 0.0 0.0 0.0 6.7 6.7 50.0

Provide security/LE services 53.3 53.3 60.0 80.0 33.3 60.0 Establish and maintain interoperable communication 60.0 40.0 33.3 50.0 40.0 50.0

Establish and maintain unified/coordinated operational structure

70.0 70.0 70.0 70.0 70.0 70.0

Update emergency operations plans 60.0 50.0 60.0 60.0 70.0 70.0

Complete triage, begin treatment, and transfer people to appropriate facilities

80.0 70.0 13.3 13.3 30.0 60.0

Deliver actionable information to people affected 80.0 80.0 65.0 50.0 65.0 50.0

Provide notification and updates to leadership and partner organizations

80.0 65.0 65.0 65.0 65.0 80.0

31

2.3.3 Assessing Impact of Benefit Categories to Overall Benefit The major benefit categories are listed in Table 5. The second column lists the weights assessed by HSSEDI in the weighted-average rollup of the category scores into an overall benefit score. The benefit category weights are relative weights assessed by the HSSEDI team based on the following rationale. These weights are subjective and should be reassessed by FEMA and its stakeholders if this methodology is implemented and used to support funding decisions. A number of methods exist for determining weights that accurately represent stakeholders’ priorities. Alternate methods for determining benefit category weights include independent or group assessments of weights by FEMA or jurisdiction Emergency Management SMEs. “Reduced risk of lives lost” carries the highest social benefit and is most highly aligned to FEMA’s preparedness goals. Therefore, HSSEDI assessed this category as the highest, a weight of “1”. The “Reduced risks of injuries/illness, property damage and business losses” were rated lower than the risk of lives lost, but higher than time spent without necessities and other QOL indicators due to the HSSEDI team’s assertion that the social/socio-economic benefits were higher for these categories.

Table 5. Major Benefit Category Contribution to Overall Investment Benefit

Category Weight

Reduced risk of lives lost 1.00

Reduced risk of injuries/illnesses 0.50

Reduced risk of property damage 0.50

Reduced risk of business losses 0.50

Reduced risk of time spent without necessities 0.25

Reduced risk of impact on other QOL issues 0.25

2.3.4 Assessing Impact of Investments to Achievement of THIRA Targets This approach includes assessing the contribution of each investment to meeting one or more of the THIRA targets using a systems dynamics simulation model, as described in Section 2.4. Starting with grant reporting data, a SME would select investments that contribute to THIRA targets. For example, grants describing search and rescue (operations) would be selected for contributing to the Mass Search and Rescue targets. With the subset of investments identified, an additional review would be completed to group them into categories based on the grant project descriptions.

2.4 Simulation HSSEDI asserted that the change in investment to generate a change in utility for the THIRA targets is likely to involve non-linear, complicated, or perhaps complex relationships. As a standard approach to resolve this utility change, HSSEDI implemented a system dynamics

32

model. This type of model allows for the collection of data about initial conditions, changes over time, and the interactions or contributions over time that characterize a system.

2.5 DOM Output The simulation has one output for each investment: the percentage change of rescues with and without adjustment rescue rates applied. This percentage change (rescue improvement) then becomes an input to the value tree structure to calculate the overall benefit score for that investment. A final roll up of benefits/risk reduction for investments is then calculated using the value tree.

2.6 DOM Pilot The DOM pilot is the implementation of the DOM model for the set of 2015 SAR investments from the State of Florida (FL). The following sections provide the implementation of the approach for the analysis of the 2015 SAR investments in Florida.

2.6.1 Review of Investments The process is described by the following steps:

1. Manually review investments for applicability – Extract FL 2015 SAR investments from BSIR by filtering by Mass Search and Rescue core capability groupings and conduct an iterative review of investments to identify a subset of investments that reference SAR capability.

2. Map investments to THIRA Targets. a. Group FL 2015 SAR investments into common categories relevant to HSGP core

capabilities using affinity analysis such as training, communication, coordination. 3. Assess the percent change/contribution of investments toward THIRA targets by

simulating investment effects on the rate of rescue by location. (Additional Input: population data by location)

4. Input results of simulation—contribution of investments toward THIRA targets—into the value tree. The simulation output is assumed to represent the utility of the investment. This value is applied across all benefit categories for the SAR THIRA target in accordance with the weights established by the SMEs (refer to Table 3).

5. Roll up benefits/risk reduction for all SAR HSGP investments using the value tree.

2.6.2 Jurisdiction Alignment to Investments Investment information was extracted from the BSIR for Florida in 2015. These data were joined with outside data sources including county-level Federal Information Processing Standard (FIPS) code and the number of fire and emergency management service (EMS) stations per 100,000 residents. Prior to the addition of these data, each investment was aligned to a county-level (or equivalent) geographic area. In the case of investments that did not align to a county-level geographic area (as a state resource), the mean value of all county-level equivalents that the

33

project can be expected to impact was used. The demographic data used in the pilot were from the 2015 American Community Survey five-year estimates, a product of the U.S. Census Bureau. The fire and EMS station data were derived from DHS’s Homeland Infrastructure Foundation-Level Data (HIFLD) for fire stations and EMS stations, combined and calculated as a per 100,000 residents value.

2.6.3 Simulation The DOM implementation for the Florida SAR investments required a means to estimate the outcome of the investments. For the pilot, the purpose of this simulation is to establish the feasibility, additional data needs, and the potential for accomplishing an estimate of this change. The DOM SAR system dynamics model was a notional representation of all Florida SAR operations for a Florida jurisdiction in 2015. Although this simplification may not address every aspect of SAR operations, it focused on the primary effects of SAR, namely the rescued population. The HSSEDI team selected the rescued population as the measure of interest for the rate of change for SAR. System dynamics models use the concepts of stocks and flows24 to document the relationships and rates of change of elements studied. Figure 6 depicts the stock and flow construct for the system dynamics model. In this figure, the stocks (Impacted, Rescued, and Loss boxes) represent the status in which portions of the population exist. The model computes changes using a time increment. For this model each time increment represents a day of real time. The transition between populations depends on rates that are determined by the size of the populations at each status and by some outside parameters (gold boxes with black arrows). Moving from left to right, after an event has occurred, some populations of people will be impacted (Impacted box) and need assistance. Each simulation time step transitions a portion of the Impacted population to the Rescued population and a portion to the Loss population.

24 For more information go to https://systemic2016.wordpress.com/system-dynamics-stock-and-flow-modelling/

https://systemic2016.wordpress.com/system-dynamics-stock-and-flow-modelling/

34

Figure 6. Systems Dynamics View

Simulation runs are compared to a baseline run where the rescue_adj = 0. The transition from Impacted to Rescued is a function of an assumed baseline performance (rate_rescue), assumed adjustments to this baseline rate given the ability of the investment being evaluated to affect this rate (rescue_adj), and a proportion of previously rescued people who are capable of volunteering (volunteer). The volunteer rate provides a feedback loop that introduces non-linear behavior. HSSEDI restricted the number of feedback loops and the complication of the model to maintain the focus on the formulation of the simulation and the application of the results to the value tree. The transition from Impacted to Loss (rate_loss) is a rate that increases over time, reflecting that the longer a population waits for rescue, the higher the rate of loss. The instantaneous transition between Impacted and Rescued at each time step is represented as follows:

Transition = rescue_rate * (1 + rescue_adj) + (volunteer * Rescued) The rescue_rate and the rescue_adj are determined for each investment. The rescue_adj is an assertion made by HSSEDI. The rescue_rate is calculated by multiplying the Fire/EMS per capita value by the number of residents in the region of the agency submitting the application divided by 100,000. The rescue_adj is the percent increase the individual investment is assessed to provide above the rescue_rate (the investment’s outcome).

2.7 Simulation Assumed Values The assumed values for the simulation were:

• Simulation time step = 1 day.

• Default impacted population that needs rescue is assumed to be 1,000. All simulation runs start with the same size impacted population and the run ends when the impacted population is exhausted.

35

• Rescue_rate – This rate is unique to each agency requesting the investment. This rate is described in Section 2.1

• Proportion of impacted population that get rescued and then re-enters the run as SAR resources (e.g., National Guard, reserves, volunteer groups) = 10 percent.

Training-related investments improve the rescue rate by 10 percent, communications-related investments improve the rescue rate by 5 percent, and coordination-related investments improve the rescue rate by 2 percent. These assumptions are based on the assertion that training is an essential enabler of successful operations, so it has the greatest impact; communications and coordination are supporting activities and have a lesser—though still important—impact. It is also assumed that these groupings are mutually exclusive. The implementation of the system dynamics modeling assumptions and rates reflect the relative scale of the representation. The model consolidates a year of events in which there are search and rescue activities into a single notional event experienced by a single notional population. In this case the notional population is 1000 people. The rescue rate used for the scenarios was 3 percent of the initial population per day and remains fixed through the simulation. The loss rate is variable to reflect the loss rate increasing with time. For the loss rate, a linear growth rate was selected linking to the time step for simplicity of analysis and explanation appropriate for a proof-of-concept exploration. Implementation of rescue rates or loss rates leveraging more complex or probabilistic frameworks may be desired in a final implementation; however, addition of this degree of resolution will add complexity to the construction and analysis. Using these rates and the computation of daily loss/rescue numbers, the systems dynamics model resolves the status of the 1000 in a modeled, three-week period. Recalling that this is a single event representing the collection of all SAR events for a year, this indicates a theoretical maximum SAR event duration at three weeks for an initial population of 1000 people. Should FEMA desire a greater resolution from the systems dynamics models, multiple models may be built using the same framework, leveraging observed, asserted, or targeted rescue rates, loss rates, and event populations. This would require the development of event models and predictions of numbers of each event by type to build a year of events to drive the loss prediction. Such models may increase confidence in system dynamics models; however, the team believes that the current level of representation is sufficient and appropriate to the input and the use of the output.

2.8 Pilot Results Table 6 presents an example of the results for a single run (Run A) of the simulation for all sixteen 2015 SAR investments. The simulation calculates the benefit of each investment using the methodology described in preceding sections. The last column (Fractional Benefit) is calculated outside of the simulation. Fractional benefit is the product of the investment benefit and the jurisdiction's fraction of the state population. For example, “City of Miami Investment 1” has a calculated investment benefit score of 0.014. The City of Miami has a population of about 2.7 million people, which is 12.6 percent of the state population of 21 million. Therefore, the fractional benefit of this investment is 0.014*0.126 = 0.0018 = 0.18 percent. The fractional

36

benefits are then summed to estimate the total benefit to the State of Florida from this set of 16 investments: 5.4 percent. The result is a relative, not absolute, valuation, useful for comparing the benefit of investment alternatives within the State of Florida (as in prioritization or rank ordering). HSSEDI assumed that the local benefits required a means to normalize prior to summary at the state level. A population-based normalization of benefit computed by benefit and by region for summary at the state level was implemented. Specifically, the fractional benefit was computed as a jurisdiction’s percentage of state population multiplied by the target satisfaction due to all investments for that jurisdiction. The maximum value that a jurisdiction’s fractional benefit could have would be equal to that jurisdiction’s fraction of the state population; this would occur when the target satisfaction produced by all investments related to that jurisdiction is 100 percent. For example, the population of the City of Miami is about 13 percent of the state population, so the maximum fractional benefit produced by all City of Miami investments would be 13 percent. The minimum value would be zero, in the case where the investment does not contribute anything to target satisfaction. The fractional benefit numbers are not very meaningful due to the limited set of investments and targets investigated in the DOM pilot (16 investments out of hundreds and one target out of 25). For the numbers to be meaningful, the entire set of investments and their impacts on all 25 THIRA targets would need to be assessed. The fractional benefit calculation for the pilot effort is provided only as an example of analysis that could be performed with a complete set of data. Should the DOM to be proven viable, the basis for this normalization would require investigation and FEMA agreement by state and amount of benefit.

37

Table 6. Example of Results from a Simulation Run

Investment Investment Name Investment Benefit Score

Fractional Benefit

City of Miami Investment 1

Support Core Capabilities Common to All Mission Areas 0.014 0.2%

City of Miami Investment 4 Mitigate the Impact of Disasters 0.027 0.3%

City of Tallahassee Investment 5

Response, Decontamination, and Rescue Operations 0.342 0.5%

City of Tampa Investment 4

Sustaining Intelligence and Information Exchange Capabilities 0.023 0.1%

City of Tampa Investment 7

Strengthening Urban Area Operational Communications Capabilities 0.055 0.3%

Alachua County Investment 7 Strengthen Communications 0.146 0.2%

Clay County Investment 3 Information and Sharing Collaboration 0.173 0.2%

Hillsborough County Investment 5


Lee County Investment 4 Regional Collaboration 0.100 0.3%

Lee County Investment 7 Strengthen Communications 0.100 0.3%

Leon County Investment 4 Regional Collaboration 0.342 0.5%

Marion County Investment 5


Miami-Dade County Investment 5


Orange County Investment 5


Florida Department of Environmental Management Investment 7

Strengthen Communications 0.087 0.1%

City of Winter Park Fire Department Investment 7

Strengthen Communications 0.041 0.2%

Total Benefit 5.4%

2.9 Scaling the Detailed Operations Model By applying the following approach and assumptions to all 2015 HSGP investments of seven states (New York, California, Texas, Illinois, the District of Columbia, Florida, New Jersey, and Pennsylvania), it is possible to assess the relative benefit of approximately 67 percent of 2015 HSGP grants:

38

• The DOM pilot considered only a tiny subset of the investments in the 2015 Florida BSIR data. Because the number of investments was so small, it was possible to conduct a manual review of individual investments to assess their applicability to the THIRA targets. For a larger set of investments, an expanded set of rules and definitions will be required to automate the process of reviewing investment data and making these assessments. Once the investments are grouped, the contribution of each investment grouping to the adjustment they make to the rate(s) would need to be assessed. The adjusted rate and the groupings were assertions that likely would need adjustments beyond this pilot.

• All BSIR records would need to be aligned to a specific geographical location. This is necessary to ensure that there is an appropriate population value available as input to the system dynamics simulation component. For the pilot, the number of investments was small enough that this could be performed by hand. For a comprehensive implementation, a geocoding script will need to be established using the agency input from the BSIR to identify the county-level equivalent geographical location and associated population and population density. Based on previous geocoding experience, HSSEDI estimates that approximately 65 percent of the BSIR records can be geocoded programmatically, while the remainder will require minimal manual effort. If implemented, the recommended improvements to BSIR data collection will make this more streamlined and decrease the need for manual work.

• The system dynamics simulation construct would need to be extended or modified to model the operations and effects of investments for each THIRA target in the value tree and for each state hazard (e.g., hurricane, fire, terrorist attack). However, it is likely that the same system dynamics simulation construct will support multiple targets and hazards. The systems dynamics model was constructed for the SAR target (Within (#) (time) of an incident, conduct search and rescue operations for (#) people requiring rescue). For each THIRA target, the rates, populations and relationships will need to be developed and constructed as additional models. Theoretically, an all-threats model may be developed in which the individual representations of threat could be exercised against a population. If an all-threats model is constructed, attention would be required in the construction of the populations and the assertions of threat exposure. Where this model is the combination of all SAR for a year on a notional population, the impacts of all threats occurring simultaneously to the same notional population will provide challenges during the post-processing step.

• One of the main challenges in the simulation effort is defining baselines and related investment effects data/rates (what rates would impact a target or outcome such as number of losses, injuries, property damage, etc.) that define interaction within the model. Simulation results would be more reflective of expected outcomes with more data and better-quality data. Each target within the value tree requires a baseline and a definition of what can improve or decrement the baseline (the adjustment rates). Once these rates have been defined, data relationships need to be identified to derive the adjustment rates for each investment.

39

2.10 Conclusions and Recommendations

2.10.1 Conclusions The DOM pilot established that development of estimates of relative investment benefit is feasible in the absence of cause-effect relationship measures, outcomes from previous investments, and inconsistent and non-standardized investment descriptions. However, the extensive use of assumptions, assertions, and SME input in place of quantifiable measures significantly diminishes the level of certainty/confidence in the result to the extent that other methodologies presented in this document may provide more meaningful results. The DOM pilot developed a generic framework, process, and implementation example that may be used either for assessment of a collection of investments or for the comparison of the investments or projects should a common, standard investment data set be developed or derived. Additionally, the model can be developed incrementally as additional data or better assessments become available. Initial results can also provide a baseline for later comparison with what could be achieved if additional data were available; establishing this baseline can support decisions about imposing additional data reporting requirements on the states. This method is best suited to comparison of alternatives, not absolute valuation. It shows the decision maker the benefit provided by an individual project (or combination of projects) and allows one project (or combination of projects) to be compared with another. Its results can therefore be used to identify gaps and potential overlaps, and the framework established under this approach sets the stage for possible future extension to portfolio analysis.

2.10.2 Recommendations HSSEDI recommends that FEMA consider the complications and issues exposed by the pilot to move to resolve the ambiguity and cross-correlation that exists in the targets, core capabilities, and investment descriptions as well as the degree to which investments contribute to the achievement of targets. Through the attempt to construct the relationships between investments, capabilities, targets, and observations of performance, the DOM pilot found that substantial work is needed to fuse the efforts into a cohesive and unambiguous network or system. The cross-correlation between capabilities and targets that has not been addressed in the lexical use of the categories will continue to obscure any attempt to apply a disciplined approach to quantifying these relationships. FEMA should consider an effort to standardize these relationships for clarity in the development of projects/investments and to provide clear direction in the estimation and measurement of performance. The SME tables and input from this effort may serve as a starting point to resolve this ambiguity; however, a potential exists that both targets and capabilities may need to be re-developed to remove the cross-correlation. HSSEDI recommends that FEMA address the data gaps and implement the data recommendations, presented in Appendix A.6.1, that resulted from the exercise of this pilot in order to improve FEMA’s ability to assess the value of the grant program in the future.

40

The value tree weights for assessing the benefit categories toward overall benefit are subjective and should be reassessed by FEMA and its stakeholders if this methodology is implemented and used to support funding decisions.

41

3 Revealed Preference Analysis of Virginia SHSP Projects This analysis estimates the value of project proposals submitted to SHSP in Virginia. Unlike a traditional CBA, this analysis does not measure the value of the SHSP project to the Virginia public. That approach was determined to be cost prohibitive. Direct approaches quantifying and monetizing the expected impact of these projects using existing data was not feasible due to the highly heterogeneous nature of SHSP grants and the lack of information on project impacts. Two alternative approaches—known in the literature as “revealed preference” and “stated preference”—infer value25 by linking actual or hypothetical behavior to project impact. Using a sample of the public to infer value for these projects is problematic because individuals cannot credibly assess risks, vulnerabilities, threats, or project impact. Therefore, revealed behavior is likely unrelated to SHSP projects, and any opinion the public may express about such projects is likely uninformed. Due to the problems inherent with the conventional approaches, this approach does not attempt to measure the public value of SHSP projects. Instead, it measures the value that SDMs in emergency management implicitly place on these projects. Section 3.1 provides an overview to this approach, while Section 3.2 discusses the data used in the pilot study. Sections 3.3 and 3.4 present the methodology and results, respectively. Finally, Section 3.5 discusses the findings and limitations of the study.

3.1 Overview Economics defines a person’s valuation of something by what that person is willing to forego to get it.26 Although this can include foregoing non-monetary items, such as time and effort, it is conventional to monetize all sources of value into a single number. The total monetized value of everything that someone is willing to give up is called “willingness to pay” (WTP). The revealed preference approach measures the state’s implicit WTP by analyzing the statistical tradeoff between project costs and other project characteristics. In short, the value of the project to SDMs is the maximum amount of funds that they would be willing to commit to that project. According to economic theory, the WTP for a certain number of goods can be derived if one knows the demand relationship (Landsburg, 2002). As shown in Figure 7, the demand curve shows how much consumers would be willing and able to pay to receive one additional good. For example, if consumers had Q*-1 goods already, then they would be willing to pay up to P* to receive Q* total goods. The sum of marginal values for all goods up to Q* reveals the WTP, or total value, of Q* goods. In other words, the area under the demand curve up to Q* indicates the value for Q* goods. The difference between this value and the costs of those goods is the net benefit to consumers. This pilot attempts to value SHSP projects in Virginia by estimating the demand of SDMs in emergency management for these projects and then calculating the relevant area under that curve to estimate their WTP. Note that SDM demand is assumed to be a function of proposal

25 Whereas revealed preference uses statistical methods to tease out the hidden preferences that govern people’s actual behavior, stated preference uses similar techniques to estimate those preferences based on peoples stated, or hypothetical behavior. 26 Landsburg, S.E., Price Theory and Applications, Fifth Edition. South Western, Cincinnati, OH, 2002.

42

characteristics; it is not simply the characteristics themselves. It is this distinction that allows a proposal’s characteristics, such as overall score, to be monetized. If the demand relationship were ignored, the benefits and costs of any proposal would be defined in different units (i.e., project score and dollars), making it difficult to assess the net gain of any funding decision.

Figure 7. Demand Curve and Willingness to Pay

Unfortunately, the federal data obtained by FEMA could not be used to estimate SDM demand at a national level for two major reasons. First, there are no variables in the existing federal datasets that represent a common basis for assessing the impact of each grant. Second, the methodology used in this section derives value by comparing the characteristics of proposals that were funded to those that were not funded, but FEMA does not collect information for unfunded proposals. Because proposal vetting and fund allocation take place at the state and local levels, state-level data can address both issues. Virginia was selected for this pilot study for several reasons. First, Virginia’s vetting process uses SMEs to independently compare all projects using a set of known and observable project characteristics. That process is designed to focus on important project qualities such as risks, hazard, and effectiveness and minimize the degree to which political considerations influence the allocation. Second, because Virginia has used a formal allocation process since 2012, enough longitudinal data exist to test whether this pilot methodology generates stable results across time. Finally, the Virginia data were convenient for MITRE to obtain, owing to pre-existing staff knowledge of Virginia’s process.

3.2 Description of Data Under the authority of the Virginia Department of Emergency Management (VDEM), the allocation of Virginia’s HSGP funds has, since 2012, followed a formal vetting process managed by the Virginia Modeling, Simulation and Analysis Center (VMASC) at Old Dominion

43

University. The process is described in detail by Ezell, Lawsure, and Flanagan (2016).27 Although the process has changed in important ways over time, the basic steps remain:

1. A multi-objective decision model is designed to reflect the priorities of the seniorleadership in the assessed project criteria. The model includes a set of questions andproject characteristics that are to be assessed by judges, as well as the correspondingweights that indicate the relative importance of each criterion.

2. Project applicants submit a written description of their project that includes why theproject is necessary, which vulnerabilities and threats it will address, the expectedimprovement, as well as how the project will be managed and evaluated.

3. Project applications are scored by SMEs using the criteria outlined in Step 1.4. Scores are aggregated using the multi-objective decision model to determine a total

benefit score. This score is divided by project cost to estimate benefit-cost ratio.5. Senior leaders receive these ratios and all other submitted information to use in deciding

which projects to fund.With permission from VDEM, VMASC provided HSSEDI with data from proposal submissions, the decision models used each year, expert scoring results, and the final award status. This was available yearly from 2012 through 2017.

3.2.1 Virginia’s Multi-Objective Decision Model Design The model used to vet each proposal requires that experts answer a series of questions, most of which are in the form of a seven-point Likert scale (i.e., completely agree, mostly agree, etc.). Each point on the Likert scale has an associated score between 0 and 100, and the score increases as the reviewer makes more favorable answers (i.e., completely agree = 100, mostly agree = 90, etc.). The scores for all questions are then weighted and summed together to make the reviewer’s total score for the proposal. Finally, the total scores for everyone reviewing a proposal are averaged to create a final score for that proposal.28 The model changes every year to reflect improvements in design as well as changing priorities of the decision makers. Changes affect the number and content of the questions, the weighting scheme associated with the questions, the scoring scheme associated with the Likert scale, and when the reviewer responses are averaged. Despite these differences, the fundamental logic of the decision model is the same across time. Table 7 shows the questions and project criteria used by reviewers to vet each proposal across all years in the sample. Because the questions change yearly, the table attempts to group the questions into similar categories. Questions and criteria about need, risk, sustainment, results evaluation, and project management were asked in all six years. Questions and criteria relating to

27 Ezell, Barry., Lawsure, K., and Flanagan, D. (2016). “Risk and Decision Analytic Support to the Commonwealth of Virginia State Homeland Security Program.” Final report submitted to the Virginia Department of Emergency Managers. 28 Prior to 2015, the Likert scale responses were averaged across reviewers prior to applying the associated score between 0 and 100 for each question. Because the associated scores increase non-linearly with the Likert scale, this methodological difference gave negative answers greater weight in the final project score.

44

regional impact and multidisciplinary or stakeholder engagement were implemented in 2015. The total number of questions in the review has also changed. In 2013 and 2014, reviewers only answered five questions per proposal. In 2015, they answered 18 questions.

45

Table 7. Questions and Question Weights Used by VDEM in the SHSP Multi-Objective Decision Model

2012 2013 2014 2015 2016 2017 Necessity / Capability Gap

How well does the organization submitting the project proposal make the case for the necessity of the project? 21% - - - - -

How well does the organization submitting the project proposal explain how this project will mitigate the risk and reduce the capability gap?

- 30% 21% - - -

The project links to core capabilities and preparedness goals. - - - 6% 6% 7% The proposal 1) identifies a capability gap, 2) the project addresses the gap, and 3) the project has letters of support. Y/N (All three must apply)

- - - - 12% 13%

RiskHow well does the organization submitting the project proposal evaluate the risk in terms of threat, vulnerability and consequence?

17% 17% 30% 11% 14% 12%

ImpactHow well does the organization submitting the project proposal make the case for mitigation efficacy? 9% - - - - -

The proposal identifies specific jurisdiction(s), the impact to the jurisdiction(s) and how the interaction occurs. - - - 5% 10% 10%

How viable are the specifics of the project plan? 9% - - - - - Sustainment

Consider whether this project sustains or enhances a current project or if it is a new project. 13% 13% 13% 9% 4% 6%

Results EvaluationHow well does the organization submitting the project proposal explain results evaluation? 19% 19% 19% 9% 2% 4%

46

Project Management Plan 2012 2013 2014 2015 2016 2017 How well does the organization submitting the project proposal explain the project management plan? 13% 21% 17% 8% 8% 3% The project manager provides timeline from grant award to completion. For SHSGP projects funded in the past three years, were they completed? Y/N/NA (Explained if no).

- - - 7% - -

Regional / Statewide Impact The project benefits the jurisdiction, region and State. The project benefits the community, region, staff and other stakeholders.

- - - 10% 13% 11%

The asset is available state wide? - - - 4% 2% 2% The Project can be easily duplicated beyond the initial scope or area of initial concern - - - 1% 1% 0%

The project has a credible plan to share resources. - - - 5% 10% 10% The proposal has documentation (Memoranda of Understanding, contracts, etc.) that demonstrate collaboration and/or agreement for multiple jurisdictions and/or region.

- - - 8% 7% 9%

The proposal has been vetted and/or endorsed regionally and documented. - - - 6% - -

If applicable, T&E is de-conflicted with State-level plan. - - - 1% - - Fiscal Stress Index - - - - - 1%

Multidisciplinary / Stakeholder Engagement The proposal addresses the degree to which the project includes multiple disciplines: fire, police departments, emergency medical services, etc.

- - - 4% 9% 8%

Letters of engagement are included to demonstrate effective coordination with affected stakeholders - - - 2% - - The project addresses for example – Public-Private Partnerships, State-Local Partnerships, access, and functional needs?

- - - 3% 5% 5%

47

The proposal is regionally screened (Y/N, NA (i.e. State proposal)); example - interoperability, sustainability, addresses special challenges, public institution of higher education.

- - - 2% - -

Multidisciplinary / Stakeholder Engagement

100% 100% 100% 100% 100% 100% • The wording on all questions have changed over time, but these changes are not reflected in the table above to help compare question weights.

• The totals may not add to the row sums due to rounding errors.

48

3.2.2 Final Sample Characteristics Based on exploratory tests, the answers reviewers gave were strongly correlated across different questions in each year, especially in the models used before 2015. For example, reviewers answered seven different questions in 2012. Even though seven questions were asked in that year, the reviewer answers were so closely correlated that there was not much new information to be had after the first question was answered. If reviewers scored one question highly, they were very likely to score all remaining six questions highly. The total number of questions increased beginning in 2015, and the answers to these questions resulted in more “new” information that was not strongly correlated with other answers. Correlations were determined through a principal component analysis done on the reviewers’ responses and found only one eigenvalue to be greater than one (see Table 8). The fact that reviewer scores are highly correlated is not necessarily a bad thing. It suggests multiple inferences: there was substantial overlap in the questions, reviewers found it difficult to distinguish between different questions when scoring the proposals, or the proposals tended to have similar qualities that impacted the scoring across all questions. However, highly correlated scores to questions means that SDM demand would be subject to multicollinearity risk if multiple questions were included in the same economic specification. Consequently, the overall score to each proposal may be the best empirical way to capture SME opinion.

Table 8. Count of Variables and Eigenvalues > 1 in Principal Components Analysis Performed on Reviewer Responses, by Year

Year Variables Eigenvalues29 >

1 2012 7 1 2013 5 1 2014 5 1 2015 18 3 2016 14 2 2017 12 3

• Beginning in 2015, questions pertaining to region were given to entirely different reviewers. Thus, thecount of eigenvalues greater than 1 comes from two separate principal component analyses conducted onthe different subsets of questions.

Table 9 lists the summary statistics from the original data set MITRE received from VMASC. The original data files contained 885 proposals over the six years. However, 137 of these proposals either did not have corresponding reviewer scores or were not part of the competitive process and were dropped from the analysis. An additional eight observations were also removed from the sample because their funding requests were outliers and greater than $1 million.30

29 A non-zero vector that changes by only a scalar factor when that linear transformation is applied to it (https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors). 30 Only one of those eight observations that requested funding over $1 million was approved. This occurred in 2013.

49

Approximately 61 percent of this sample was approved by SDMs, but that rate varied over time. The average proposal scores also varied considerably over time, but approved proposals received higher scores than proposals that were not approved, on average. Similarly, the average proposal that was approved requested less funding ($49,354) than the average proposal that was not approved ($167,845), but the annual averages changed between years.

Table 9. Counts and Mean Characteristics of Sample Observations 2012 2013 2014 2015 2016 2017 Original Sample 112 164 185 155 143 139 Less Missing Score / Not SHSP 8 0 0 0 37 31 Less Partially Missing Scores 0 0 0 61 0 0 Less Request > $1 million 1 3 2 1 1 0 Final Sample 103 161 183 93 105 108 Sample by Approval Status Not Approved 36 69 71 35 37 43 Approved 67 92 112 58 68 65 Total 103 161 183 93 105 108 Average Proposal Score Not Approved 31.5 21.4 35.4 65.6 69.8 58.9 Approved 33.7 58.3 40.4 74.3 71.2 70.4 Total 32.9 42.5 38.5 71.0 70.7 65.8 Average Funding Request Not Approved $131,267 $167,822 $203,183 $177,991 $177,441 $123,640 Approved $53,123 $69,656 $42,724 $48,624 $39,802 $38,802 Total $80,435 $111,727 $104,978 $97,310 $88,303 $72,580

The information in Table 10 provides a limited picture of the underlying VMASC data used in this study. Table 10 shows the total size of requested and awarded funds by year and also decomposes the proposals into those that were fully vetted by SMEs and allocated competitively and those that were not. The table reveals that VA competitively awarded between $2.5 and million $2.9 million in 2012, 2015, 2016, and 2018. In 2013 and 2014, however, VA awarded $5.1 million and $4.8 million competitively. These differences suggest the selection process may have differed enough in these two be cautious when drawing inferences between years. Information on those proposals that were not fully vetted by SMEs and/or competitively allocated are presented in the two columns on the right in Table 10. The data show that all proposals in 2013 and 2014 were competitively allocated, but that between a third and a half of the requested funds were not competitively allocated beginning in 2015. Overall, the information in the table also suggests that the total number of requests for funds has generally been falling over time. In 2013, $22.6 million in funding was requested. In 2017, only $12.2 million in funding was requested.

50

Table 10. Total Amount of Requested and Awarded Funding, by Competitive Process

Scored Proposals / Competitive Process

Unscored Proposals / Noncompetitive Process

Total Requested

Funds

Total Approved

Funds

Total Requested

Funds

Total Approved

Funds 2012 $10,384,838 $2,886,368 $1,296,028 $450,000 2013 $22,641,055 $5,093,584 $0 $0 2014 $22,876,231 $4,785,065 $0 $0

2015* $10,679,666 $2,820,190 $6,708,351 $0 2016 $9,271,847 $2,700,821 $6,780,897 $2,692,156 2017 $7,838,604 $2,522,099 $4,325,823 $2,671,927

• While the data received by VDEM suggests that all non-competitive proposals were rejected in 2015, it is possible that these proposals were funded by alternative sources.

For improved clarity, Figure 8 shows the distribution of the final proposal scores. Project scores had a substantial variability every year, and there is considerable overlap in the scores between approved and unapproved projects.

51

0

20

40

60

80

100

Approved Not Approved

2012

0

20

40

60

80

100


2013

0

20

40

60

80

100


2014

0

20

40

60

80

100


2015

0

20

40

60

80

100


2016

0

20

40

60

80

100


2017

Figure 8. Distribution of Final Proposal Scores, by Year and Approval Status

Figure 9 shows the distribution of the requested funds. The figures indicate that, in general, less expensive projects are much more likely to be approved. Prior to 2015, some projects were approved that requested much larger sums than approved projects in later years. For example, 2012 and 2013 were the only years in which projects over $250,000 were ever approved. In addition, the approved projects have considerably less variability than the unapproved projects. Since 2015, no approved project had requested funding above $115,000.

52

• Proposals that requested over $1,000,000 are not shown.

Figure 9. Distribution of Proposal Requested Funds, by Year and Approval Status

The distributions in Figure 9 suggest applicants have, on average, reduced their funding requests over time. This could be because applicants learned that projects with larger funding requests have a relatively small chance of being selected, even if they have relatively high project scores. This would be expected if project selection is primarily based on each project’s score and funding request combined in a benefit-cost ratio (where the score is the measure of benefit). Because project scores have a maximum value of 100 points and each additional point affects the ratio of proposal score to proposal cost by the same amount,31 proposals may simply differentiate themselves by requesting less funding. For example, a project that receives a score of 100 and requests $100,000 in funding would have a lower benefit-cost ratio (0.001) than a project that receives a score of 100 and requests $10,000 (benefit-cost ratio of 0.01). Similarly, projects with lower scores and lower funding levels can have the same or better benefit-cost ratios as projects with higher scores and higher finding levels.

31 Specifically, each additional project score raises the ratio by a magnitude of 1/funding request.

53

From the data, it is unclear to what extent the pattern of funding lower-cost projects is the result of the evaluation process or the presence of additional information known to the SDMs. For example, the experience of the SDMs may be that smaller projects tend to produce better, more consistent results, or there may be a political preference for smaller projects to allow the HSGP funding to be spread across more localities.

3.3 Methodology Logistic regression was used to estimate the implicit demand curve of SDMs for projects. The technique assumes that the SDMs have some utility dependent on both the project scores, the requested funds, and a component that is not observable in the data. To be specific, the technique assumes that the utility of the ith project, Ui, is a linear combination of the proposal’s score, Si, and its cost, Ci. However, the SDMs may also use other unobserved information when deciding whether they should fund a project. That unobserved component is assumed to be random and enters the SDMs’ utility as ɛi. The utility that the ith project brings to the SDMs is represented as follows:

𝑈𝑈𝑖𝑖 = 𝛼𝛼1 + 𝛼𝛼2 × 𝑆𝑆𝑖𝑖 + 𝛼𝛼3 × 𝐶𝐶𝑖𝑖 + 𝜀𝜀𝑖𝑖 Next, it is assumed that if Ui>0, the project will be funded. And although utility is never observed, logistic regression can be used to estimate α1, α2, and α3 if ɛi is assumed to be independently32 and identically distributed within the extreme value distribution. In this case, the probability that SDMs fund the ith proposal is:

𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃(𝑈𝑈𝑖𝑖 > 0 |𝑆𝑆𝑖𝑖 ,𝐶𝐶𝑖𝑖) =𝑒𝑒𝑒𝑒𝑒𝑒(𝛼𝛼1 + 𝛼𝛼2 × 𝑆𝑆𝑖𝑖 + 𝛼𝛼3 × 𝐶𝐶𝑖𝑖)

1 + 𝑒𝑒𝑒𝑒𝑒𝑒(𝛼𝛼1 + 𝛼𝛼2 × 𝑆𝑆𝑖𝑖 + 𝛼𝛼3 × 𝐶𝐶𝑖𝑖)

Although it may not be immediately intuitive, the probability that SDMs approve the ith proposal is defined as the “quantity” that SDMs demand for that proposal. To understand the connection, suppose that SDMs reviewed 10 proposals that all had the same score and cost. Further suppose that the logistic regression predicts that any one proposal with these features has a 40 percent chance of being approved. Given these assumptions, the SDMs can be expected to approve 4 of the 10 proposals. Equivalently, 40 percent can be interpreted as the total proportion of proposals with the same score and cost that one expects will be approved. To formalize this intuition, suppose there are a total of ‘ni’ proposals that have the same characteristics as the ith proposal. The number of approved proposals with characteristics matching the ith proposal, Qi, is expected to be:

𝐸𝐸[𝑄𝑄𝑖𝑖 | 𝑆𝑆𝑖𝑖 ,𝐶𝐶𝑖𝑖 ,𝑛𝑛𝑖𝑖] = 𝑛𝑛𝑖𝑖 × 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃(𝑈𝑈𝑖𝑖 > 0 |𝑆𝑆𝑖𝑖 ,𝐶𝐶𝑖𝑖)

The two previous equations can be combined to show the maximum price that SDMs would be willing to pay to approve the marginal ith project. In other words, they can show the height of the

32 The assumption of independent errors terms does not hold in the cost allocation process because SDMs have a fixed budget to allocate across different projects. If one project is selected, this reduces the chances of other projects being selected. However, to demonstrate the viability of this pilot, this violation of this model assumptions is ignored. Future work will be required to ensure that the estimation procedure accounts for this correlation between error terms.

54

demand curve, Ci, as a function of quantity, 𝐸𝐸[𝑄𝑄𝑖𝑖 ], proposal score, Si, and the number of observationally equivalent proposals being considered by SDMs, ni.

1𝛼𝛼3

× 𝑙𝑙𝑛𝑛𝐸𝐸[𝑄𝑄𝑖𝑖| 𝑆𝑆𝑖𝑖 ,𝐶𝐶𝑖𝑖 ,𝑛𝑛𝑖𝑖 ]

𝑛𝑛𝑖𝑖 − 𝐸𝐸[𝑄𝑄𝑖𝑖| 𝑆𝑆𝑖𝑖 ,𝐶𝐶𝑖𝑖 ,𝑛𝑛𝑖𝑖 ]− 𝛼𝛼1 − 𝛼𝛼2 × 𝑆𝑆𝑖𝑖 = 𝐶𝐶𝑖𝑖

Figure 10 shows a generic demand curve which assumes there are ten hypothetical proposals with the same score under evaluation by SDMs.

0

50,000

100,000

150,000

Prop

osal

Cos

t

0 2 4 6 8 10Quantity of Approved Proposals

Figure 10. Heuristic SDM Demand for 10 Proposals that All Have Matching Scores and Costs

Similarly, each curve in Figure 11 represents SDM demand for unique projects that have different scores. The blue curve represents the demand for a proposal with an overall score of 65, while the red curve represents demand for a proposal with a final score of 85. It can be shown that the difference in height between these curves equals the difference in project scores multiplied by -α2/α3. This ratio represents the SDMs’ implicit WTP per point in a proposal’s score. If the logistic regression finds that SDMs prefer proposals with high scores and low costs, this ratio will be positive.

55

Figure 11. Heuristic Demand Curves for Two Proposals Wtih Different Reviewer Scores

Recall from Section 3.1 that WTP is the area under the demand curve over the range of relevant quantities. In the special case where a project either has a unique score or is being evaluated independently, 𝑛𝑛𝑖𝑖 = 1. In this instance, the only relevant quantities that matter are 0 (i.e., proposal is not approved) and 1 (the proposal is approved). In other words, the SDMs’ willingness to pay for the ith proposal, WTPi, is the integral under the demand curve from 0 to 1.

𝑊𝑊𝑊𝑊𝑃𝑃𝑖𝑖 =1𝛼𝛼3

× 𝑙𝑙𝑛𝑛𝐸𝐸[𝑄𝑄𝑖𝑖]

1 − 𝐸𝐸[𝑄𝑄𝑖𝑖] − 𝛼𝛼1 − 𝛼𝛼2 × 𝑆𝑆𝑖𝑖 𝑑𝑑𝐸𝐸[𝑄𝑄𝑖𝑖]

which reduces to a much simpler function:

𝑊𝑊𝑊𝑊𝑃𝑃𝑖𝑖 =−(𝛼𝛼1 + 𝛼𝛼2 × 𝑆𝑆𝑖𝑖)

𝛼𝛼3The net benefit to SDMs for approving the ith proposal is therefore what they would be willing to pay to approve the ith proposal minus that proposal’s requested funds: 𝑊𝑊𝑊𝑊𝑃𝑃𝑖𝑖 − 𝐶𝐶𝑖𝑖. The total net benefit of the SHSP program to SDMs is the sum of net benefits across all approved projects.

56

3.4 Results Table 11 shows the results of the logistic regressions used to estimate the probability that a given proposal would be approved. The results confirm the prior expectations that the chances of getting a proposal approved increases with the final score of reviewers and decreases with the requested funds. The model also found these relationships to be highly statistically significant, which is to be expected given VDEM’s formal decision-making process. The only exception to this is the 2012 model, in which reviewer score was not statistically significant.33

33 This may be expected since 2012 was the first year that implemented a formal review process, and the design of the multi-objective decision model was overhauled in 2013. The low pseudo-R2 value of 0.1 also suggests a poor model fit in 2012.

Table 11. Coefficient Estimates from Logistic Regressions of Proposal Approval Status, by Year

2012 2013 2014 2015 2016 2017 Constant 0.8519 -3.8279 *** -0.0322 -6.7689 ** -0.4520 -2.0217

Final Score 0.0121 0.1185 *** 1.0426 ** 0.1393 *** 0.1015 ** 0.1260 ***

Requested Funds -7.50E-06 ** -6.28E-

06 ** -4.397E-04 ** -3.221E-

05 *** -8.772E-05 *** -9.489E-

05 ***

N 103 161 183 93 105 108 Pseudo-R2 0.100 0.611 0.943 0.476 0.622 0.668

• Statistically significant at the 1 percent (***), 5 percent (**), and 10 percent (*) levels

Recall from Section 3.3 that the WTP per point in a proposal’s final score can be obtained by taking the ratio of the coefficients for final score and requested funds. Table 12 shows these estimated ratios and corresponding statistical precision across all years. The estimates suggest that SDMs implicitly valued each increase in a proposal’s scored point by a dollar value of $1,328 in 2017. However, the 95 percent confidence interval of this ratio falls between $807 per point and $1,723 per point. The point estimates and confidence intervals for WTP per point in 2016 were similar to their respective values in 2017.

Table 12. WTP per Point, by Year

Year WTP per

Point Standard

Error 95% Confidence Interval

Min Max 2012 $1,615 $1,977 -$2,259 $5,489 2013 $18,865 $6,698 $5,738 $31,993 2014 $2,371 $118 $2,139 $2,603 2015 $4,323 $1,098 $2,172 $6,475 2016 $1,158 $288 $592 $1,723 2017 $1,328 $266 $807 $1,848

57

Over time, the estimated WTP per point has changed significantly, but this is not surprising given the major changes in the formal process of selection. On the other hand, the WTP estimates for 2012, 2013, and 2015 are much less precisely estimated than the other three years. This implies that the valuation for all projects in these years were subject to greater degrees of uncertainty relative to the proposals offered in 2014, 2016, and 2017. The final step in estimating SHSP’s total value to Virginia’s SDMs is to calculate the WTP for every project using the last formula of Section 3.3 and comparing that to total requested funds. This is done in Table 13. Columns 2 through 5 indicate the aggregate value, cost, net benefit, and benefit to cost (B/C) ratio to SDMs for proposals that were not approved each year. The four columns on the right report the same information for approved proposals. In aggregate, Virginia’s SDM behavior suggests that they receive between 1.86 to 2.24 times as much value from the SHSP programs than the total amount of allocated funds. The table also suggests that the rejected proposals would have given them less value than the project costs had they been chosen, with B/C ratios between 0.41 and 0.46.

Table 13. Aggregate Proposal Benefits and Costs by Year and Approval Status

Not Approved Approved

Year Value Cost Net Benefit B/C Ratio Value Cost Net Benefit B/C

Ratio 2012* $5,918,936 $4,725,599 $1,193,337 1.25 $11,257,449 $3,559,239 $7,698,210 3.16

2013* -

$14,215,389 $11,579,700 -

$25,795,089 n/a $45,217,411 $6,408,306 $38,809,105 7.06 2014 $5,960,095 $14,425,972 -$8,465,877 0.41 $10,718,301 $4,785,065 $5,933,236 2.24

2015* $2,571,225 $6,229,674 -$3,658,450 0.41 $6,435,022 $2,820,190 $3,614,833 2.28 2016 $2,799,980 $6,565,327 -$3,765,347 0.43 $5,256,581 $2,706,520 $2,550,060 1.94 2017 $2,446,203 $5,316,505 -$2,870,302 0.46 $4,695,135 $2,522,099 $2,173,037 1.86

* Due to modelling issues, the WTP estimates were imprecisely estimated in 2012, 2013, and 2015.

58

Figure 12. Distribution of Individual Proposal Net Benefits ($) by Year and Approval Status

Figure 12 shows the distribution of net benefits across all SHSP proposals in the sample. The models indicate that the type of SHSP projects used in Virginia captures relatively small amounts of net benefits, particularly in later years in the sample. Proposals that were not funded, on the other hand, had a wide range of values that typically would have had more costs than benefits.

3.5 Discussion The purpose of this study was to evaluate whether one could use the actual allocation of program funds to infer program value. Virginia’s allocation of SHSP funds was used as a pilot study. The study suggests that it is technically feasible to construct a valuation, and initial estimates suggest a B/C ratio of approximately 2. However, it should be emphasized that this valuation pertains to decision makers, not the public. Because SHSP funds are ultimately borne by the public through higher taxes, an ideal CBA would estimate the public’s value of these grants. Unfortunately, the difference between the public’s value and the decision makers’ value is not fully understood. Donahue, Eckel, and Wilson (2013) may provide some degree of insight.34 Their study had decision makers and the public play a game in which players made a choice that bestowed different levels of risk and reward. They found that when public officials make a

34 Donahue, A., Eckel, C., and Wilson, R., “Ready or Not? How Citizens and Public Officials Perceive Risk and Preparedness.” American Review of Public Administration, 44(4), 2013, pp 89-111.

59

choice on behalf of the public, they are likely to impose their own preferences. If Virginia’s SDMs are more risk averse than the public they serve, the results of the game suggest that the public may value SHSP grants by less than what has been calculated in this pilot. Of course, more work would be needed before a more definitive comparison can be made between SDM preferences and the preferences of the public they serve. Although four of the six years in the Virginia data suggest a B/C ratio within a narrow range, it would be a mistake to assume that the same B/C ratio applies to other states and other programs. As a central, eastern seaboard state, Virginia’s needs and project proposals likely differ from most other states. Because Virginia’s funding process emphasizes the ratio between scores and cost, and scores are capped at 100, proposals may attempt to stand out by reducing requested funds instead. If so, then competition amongst applicants may be driving down the available price that SDMs implicitly pay for each scored point over time. And since price reflects marginal value, that competition would be reflected in lowering the SDM valuation over time. In addition to the possibility that SDMs in other locales have different preferences, they may also utilize a different allocation technique that does not drive down valuation. If program managers wanted to expand this pilot study, some important points should be considered.

1. The local allocation process for program funds must collect data on all proposals that were considered. There must be enough proposals that were both selected and not selected so that the model can predict which variables matter most to decision makers.

2. All proposals must be vetted primarily by evaluating common quantifiable and/or categorical characteristics. For example, the value of Virginia’s SHSP proposals was primarily based on the answers to five to 18 questions that were common to all proposals.

3. It is necessary to obtain proposal cost and award status to infer value. 4. Statistical models are not guaranteed to meet one’s prior expectations. It is possible, for

example, that a model could find a positive relationship between demand and project cost. In addition, different types of statistical models may be necessary depending on the process by which each locale selects the winning proposals.

3.5.1 Limitations It should be noted that some of the statistical assumptions required by the logistic regression model are unlikely to hold. Errors are often correlated across observations since the selection of one award reduces the chances of other awards being selected. Additionally, the funding request is likely to be an endogenous covariate if applicants lower their requested funding to increase the probability of grant selection. However, this pilot study uses the logistic regression model to demonstrate the feasibility and type of output that can be generated by the revealed preference approach. If FEMA deems that this pilot study should be expanded, more work will be needed to ensure that the econometric estimator can address these important statistical issues. Another limitation is that the WTP formula in Section 3.3 values every grant using the estimated marginal rate of substitution between points and cost. If SDMs have diminishing marginal utility with respect to points, for example, then it would be inappropriate to apply a marginal concept in

60

the valuation of an entire grant. On the other hand, since the B/C ratio is emphasized during the allocation process, the assumption of constant marginal value may actually be appropriate for Virginia. The final limitation to note for this pilot study is that the specification was extremely sparse. SDMs could take other known information into account when deciding whether to fund proposals. Other control variables like region, investment area, and how the funds were to be spent (e. equipment vs non-equipment) were included in unreported models, but none of these variables improved model fit or were theoretically necessary to include considering VDEM’s formal process. Different model specifications may be appropriate for processes used by other states.

3.5.2 Recommendations Should FEMA decide to pursue the revealed preference approach, there are recommendations that should be considered for moving beyond the pilot phase. First, the model specification in this pilot was as sparse as possible, but future specifications might allow SDM demand to depend on other factors as well. For example, the model could potentially include controls for the size of the population affected by the proposal, equity in the allocation process, whether the proposal builds a new capability or maintains an existing one, the importance of past decisions to fund a project, etc. The advantages of making SDM demand a function of multiple project characteristics is that it creates a better fit of the data and a more nuanced understanding of SDM priorities. However, such efforts are not expected to move the final B/C ratio in any specific direction.35 As mentioned in the previous section, this pilot study ignored some important statistical and econometric issues that come with demand estimation. For example, correlation in the error terms between different observations can bias coefficients estimated via logistic regression, but this is not the case with ordinary least squares regression. Therefore, future work may wish to estimate a linear probability model as a check on the degree to which that correlation may be affecting proposal valuation.36 Another statistical issue that should be explored if this pilot is expanded in the future is the assumption that funding requests are exogenous to the funding decision. If applicants lower their funding request to increase the chances of being selected, then the funding request may be creating endogeneity bias in the demand specification. If so, then an identification strategy, such as instrumentation, would be necessary. Finally, prior to expanding this pilot study, all states and urban areas should be canvassed to determine which ones allocate HSGP funds through a formal process as Virginia does. While Virginia was an ideal choice for conducting this pilot, the prevalence of their formal vetting and

35 In unreported results, some of the expanded model specifications were tested, but no consistent pattern or inferences could be drawn. Multicollinearity plagued those specifications that directly incorporated SME scores to specific questions. However, it is possible that data from other states and urban areas could allow for a richer model specification. 36 The linear probability model is the application of ordinary least squares regression on a binary dependent variable. While the disadvantages of running these models are well known (incorrect standard errors, nonsensical probability predictions for observations far from the sample mean, etc.), they can be advantageous to use in certain circumstances, such as when errors are correlated with each other or when heteroskedasticity is present.

61

allocation process is unknown. If a relatively low portion of HSGP funds are competitively awarded, then expanding the pilot would create cost-benefit ratios that are relatively uninformative of the overall HSGP program. On the other hand, if data exist for a sizeable portion of HSGP awards that can be used to infer SDM value from a formal vetting process, then expanding this pilot could provide meaningful insight.

62

4 Breakeven Analysis The focus of the BA approach is on national core capabilities and how much they reduce losses from terrorism under the Prevention Portfolio. A model was employed to generalize the effect of groups of HSGP funded projects, with each group representing a core capability towards a national capability. After analyzing the connection between project activities and their outcomes, SMEs translated outcomes to loss reductions (e.g., reduction in loss of life due to terrorist attacks). When paired with the costs of such projects, the model produced a Net Benefit Cost Ratio (NBCR) range at the core capability level.

4.1 Overview Human-caused threats and hazards occur rarely; thus, there are insufficient data to employ robust statistical methods to determine with relative certainty the likelihood of successful attacks. The rare nature of these events, combined with the national-level generalization of the model, points to a need for a model that can handle uncertainty as a key component in the computation of the NBCR. The data pulled from the Global Terrorism Database (GTD) provides a framework for the types of effects that can be reduced (e.g., the frequency of events, the conditional probability of events causing damage, the conditional damage). At the general level, the effects align with the five mission areas of the National Preparedness Goal:

• Prevent (to reduce frequency of events)

• Protect (to reduce the conditional likelihood that events can cause damage)

• Mitigation, response, and recovery (to reduce and recover from the consequences of events when there is damage)

Although HSSEDI can compute the baseline terrorism threat/hazard exposure and knows the annual FEMA expenditure in capabilities intended to reduce terrorism, HSSEDI does not have the information about how the investments improved the capabilities in communities or how those capabilities reduced the annual threat/hazard exposure. When data are not available to yield probabilities needed for a model, SMEs can provide these probabilities. However, the experts’ probability judgments must be quantitative to monetize the values.

63

4.2 Approach The BA approach used individual grants to local governments as surrogates to describe a national core capability aimed at reducing the threat of a man-made event and/or reducing the magnitude of the effect of an event (e.g., fatalities, injuries, and property damage).37 The GTD was used to build the model, including determining the conditional and non-conditional probabilities built into the model. All the project and core capability38 information, including core capability budgets, is found outside of the model. The model generates the total space of NBCR possibilities over the range of 1 percent to 100 percent for reduction of impact and over the range of 1 percent to 100 percent for reduction of attack frequency. The SMEs then determined which intervals within those all possible ranges a particular project represents. In other words, the SMEs narrowed the ranges from everything possible (1 percent to 100 percent) down to specific smaller intervals of reductions in impact and reductions in attack frequency that correspond to particular NBCRs. This method treats all 56 states and territories as a single entity in order to produce a national-level model. Individual projects are rolled up to national core capabilities for incorporation into an NBCR model (the BA model). The model, and the SME elicitation described in this section, served to generalize project outcomes and their effects to the national level, rather than attempting to identify the exact marginal benefit of specific projects. The sampling objective was to find the fewest number of projects under each core capability in the Prevention portfolio that would: 1) satisfy all the key characteristics (FEMA’s term is “critical tasks”)39 of the core capability; and 2) ensure that each project selected was representative of similar projects that were implemented in other jurisdictions or states. When these two criteria were met, a national-level core capability was determined to have been captured even though the projects selected were not implemented in every state of the United States. The approach scaled well since it did not require a unique model for every locally implemented individual project. HSSEDI developed a prototype tool that aided identifying ranges of possible outcomes in terms of positive, breakeven, and negative NBCR using the GTD for the threat profile as well as

37 Initially, the SMEs were asked to assess each of the core capabilities directly, without looking at any projects at all. However, after some candidate SMEs reviewed the core capabilities, they said they were too general and abstract to assess directly in terms of their contribution to reducing national terrorism losses. It was clear that it was necessary to go one level deeper, namely to the project level. Rather than trying to assess the many hundreds or thousands of projects under a national Core Capability, Adam Rose’s approach was followed, and project samples were drawn. If a sample of projects could capture all of the key characteristics of a Core Capability, then the sample would represent a national Core Capability. (Rose et al., “Benefit-Cost Analysis of FEMA Hazard Mitigation Grants,” DOI: 10.1061/(ASCE)1527-6988-(2007)8:4(97)). 38 There are three core capabilities that cross all portfolios. These are planning, public information and warning, and operational coordination. These were not included in Prevention Portfolio as the core capabilities are included in all portfolios and are too broad for application in a specific portfolio. 39 U.S. Department of Homeland Security, National Prevention Framework, Second Edition. June 2016. [https://www.fema.gov/media-library-data/1466017309052-85051ed62fe595d4ad026edf4d85541e/National_Protection_Framework2nd.pdf]

64

authoritative published cost estimates associated with various consequences of these threats. Inputs to the tool are mean funding levels of each of the core capabilities over a period. The core capabilities were selected to represent the Prevention Portfolio. The SMEs40 made determinations of how much the core-capability investments reduce losses due to terrorism. To provide these inputs HSSEDI formed two teams41 consisting of four SMEs on terrorism prevention and physical security (see Appendix B.3). Before the elicitation session, HSSEDI conducted elicitation training42 to improve the uncertainty calibration of the experts, consistent with the work of Cooke and Gossens (2008).43 The goal of the SME calibration was to limit bias and enhance the confidence with which SMEs provided their estimates. The calibration increased the likelihood that the true values of the core capability’s capacity to reduce the frequency and impacts of terrorist attacks were within their confidence range, and that the estimated confidence range was as narrow as possible.44

40 The MITRE Corporation employs approximately 240 counter-terrorism experts. This formed the pool from which the SMEs were selected. These experts are spread across the corporation in various centers such as the Homeland Security Center, the Center that supports DOJ/FBI, and National Security Sector programs that support the Intelligence Community. 41 One MITRE Terrorism Prevention SME participated in both rounds. 42 The calibration questions were written by HSSEDI’s Gregory Chesterton, a Multidisciplinary Systems Engineer in the MITRE-operated Center for Advanced Aviation Systems Development. Mr. Chesterton is a long-time practitioner of expert calibration and elicitation who follows the guidelines of the leader in the field, Douglas W. Hubbard. There were 48 calibration questions spread over three iterations of calibration. The first set of calibration questions were in the domain closest to the experts’ knowledge, namely counter-terrorism. Hence, those in the first set were very close in content to the target set of questions asked during the project elicitation session. The second and third iterations were in domains far afield from terrorism and hence far afield from the target set of questions. 43 Cooke, R.M. and Gossens, L.L.H.J., TU Delft expert judgment data base. Reliability Engineering and System Safety. 93(5):657-674, 2018. This information was incorporated into a MITRE calibration training briefing, “Calibrating the Human Instrument: Quantitative Estimation from Human Experts”, 6 June 2018. 44 To determine what an appropriate confidence level within this method, HSSEDI contacted Hubbard Research and spoke directly to its founder Dr. Douglas Hubbard. Dr. Hubbard stated a reasonable expectation for calibration from individuals who take his Webinar training is an 80 percent to 90 percent calibration level. This corresponds to a statistical confidence level. For example, if a 90 percent calibrated SME were given 20 projects to assess, then that student should get 18 out of 20 correct answers. That means that his low to high uncertainty ranges would contain the correct answer about 90 percent of the time. According to Dr. Hubbard, this information can be found in Chapter 7 of his book, How to Measure Anything.

65

4.3 BA Process Flow The process flow includes the SME elicitation session as well as the BA model. The BA process steps are shown in Figure 13.

Figure 13. BA Process Flow

• An elicitation session moderator presented projects45 under each of the core capabilities to the SMEs for consideration. The project was described by the session moderator to the extent that the BSIR and Investment Justification (IJ) data allowed. SMEs were then able to ask clarifying questions and discuss expected impacts.

• The SMEs entered their assessment results into the SME elicitation worksheet for each project. The assessment of each project includes the degree to which, in percentage terms, the project is likely to result in a reduction in the probability of a terrorist attack as well as reduce the probabilities of impacts of an attack.

• The SME assessments of the HSGP projects were tabulated and rolled up into each of the Prevention core capabilities to get national-level NBCR ranges. The national-level NBCR ranges were determined by applying the BA model to the rolled-up SME assessments. The core capabilities were rolled up to the Prevention Portfolio-level to provide a single NBCR range for the entire portfolio.

45 Project samples had to meet two key criteria: 1) each project had to be replicated in multiple states to realize the nationwide capability that the core capability under which it was categorized represents; 2) the project sample had to satisfy all the key characteristics of the core capability from which the sample was drawn.

66

4.3.1 Net Benefit Cost Ratio Model

4.3.1.1 Computing Net Benefit Cost Ratio To determine the effectiveness of the HSGP, HSSEDI employed the following methodology to determine the NBCR. This method demonstrated that the probability of an outcome (e.g., reduction in losses from terrorism) can be decomposed into root probabilities and conditional probabilities. In the case of a loss stemming from an adversarial hazard, its probability can be decomposed into the product of the probability of the threat (root probability), the probability of a hazard given a threat (conditional probability), and the probability of a loss given a hazard (conditional probability). For each core capability, the following was defined:

𝑵𝑵𝑵𝑵𝑵𝑵 𝑩𝑩𝑵𝑵𝑩𝑩𝑵𝑵𝑩𝑩𝑩𝑩𝑵𝑵 = 𝑩𝑩𝑵𝑵𝑩𝑩𝑵𝑵𝑩𝑩𝑩𝑩𝑵𝑵 –𝑪𝑪𝑪𝑪𝑪𝑪𝑵𝑵 and

𝑵𝑵𝑵𝑵𝑵𝑵 𝑩𝑩𝑵𝑵𝑩𝑩𝑵𝑵𝑩𝑩𝑩𝑩𝑵𝑵 𝑪𝑪𝑪𝑪𝑪𝑪𝑵𝑵 𝑹𝑹𝑹𝑹𝑵𝑵𝑩𝑩𝑪𝑪 = (𝑩𝑩𝑵𝑵𝑩𝑩𝑵𝑵𝑩𝑩𝑩𝑩𝑵𝑵 –𝑪𝑪𝑪𝑪𝑪𝑪𝑵𝑵) / 𝑪𝑪𝑪𝑪𝑪𝑪𝑵𝑵 Cost, the amount of HSGP funding applied to the core capability, is known: it is the average annual budget over five years (no discount rates were applied). To decompose Benefit into quantities that can be determined or estimated, the following is asserted:

𝑩𝑩𝑵𝑵𝑩𝑩𝑵𝑵𝑩𝑩𝑩𝑩𝑵𝑵 = 𝑷𝑷𝒓𝒓𝑵𝑵𝒓𝒓𝒓𝒓𝒓𝒓𝑵𝑵 ∗ 𝑳𝑳

L is the total annualized terrorist attack loss to be avoided. This is a constant value: $6.95 billion ($1.48 billion [monetized fatalities/injuries] + $5.47 billion [monetized property damage]) obtained from the GTD. L represents annual U.S. losses from all types of terrorist attacks for 20 years from 1995- 2015, including the 9/11 attack. The Department of Transportation (DOT) provided the data for the statistical value of life.46 (See the entries in Table 15 corresponding to Baseline Annual U.S. Exposure to Terrorism (BTR): Property Damage and Baseline Annual U.S. Exposure to Terrorism (BTR): Fatalities/Injuries.) Preduce is the probability that a particular core capability will reduce the losses defined by L. Thus, Benefit is the monetized output of determining how likely a core capability is to reduce losses from terrorist attacks multiplied by total losses from the attacks (L). Preduce can be further decomposed to reflect the conditional probabilities described above:

𝑷𝑷𝒓𝒓𝑵𝑵𝒓𝒓𝒓𝒓𝒓𝒓𝑵𝑵 = (𝑷𝑷𝑻𝑻 ∗ 𝑷𝑷𝑯𝑯|𝑻𝑻) ∗ 𝑷𝑷𝑳𝑳|𝑯𝑯

(PT * PH|T) is the probability of reducing the threat (PT) multiplied by the probability of reducing the hazard, given the threat (PH|T). In other words, this is the reduction in the likelihood of a successful attack. The second term, PL|H, is the conditional probability of reducing a loss, given the hazard. This term represents the reduction of likelihood of an impact (due to an attack). It is these two probabilities that SMEs are asked to evaluate in the BA elicitation process. Substitution into the Net Benefit equation yields:

46 U.S. Department of Transportation (DOT) Policy Guidance from 2016.

67

𝑵𝑵𝑵𝑵𝑵𝑵 𝑩𝑩𝑵𝑵𝑩𝑩𝑵𝑵𝑩𝑩𝑩𝑩𝑵𝑵 = (𝑷𝑷𝑻𝑻 ∗ 𝑷𝑷𝑯𝑯|𝑻𝑻) ∗ 𝑷𝑷𝑳𝑳|𝑯𝑯 ∗ 𝑳𝑳 –𝑪𝑪𝑪𝑪𝑪𝑪𝑵𝑵

Combining Net Benefit and Cost from n core capabilities would allow determination of the Net Benefit Cost Ratio for a Portfolio:

𝑵𝑵𝑵𝑵𝑵𝑵 𝑪𝑪𝑪𝑪𝑪𝑪𝑵𝑵 𝑩𝑩𝑵𝑵𝑩𝑩𝑵𝑵𝑩𝑩𝑩𝑩𝑵𝑵 𝑹𝑹𝑹𝑹𝑵𝑵𝑩𝑩𝑪𝑪 =∑ 𝑵𝑵𝑵𝑵𝑵𝑵 𝑩𝑩𝑵𝑵𝑩𝑩𝑵𝑵𝑩𝑩𝑩𝑩𝑵𝑵𝑩𝑩𝑩𝑩𝑩𝑩=𝟏𝟏∑ 𝑪𝑪𝑪𝑪𝑪𝑪𝑵𝑵𝑩𝑩𝑩𝑩 𝑩𝑩=𝟏𝟏

The core capabilities that represent the Prevention Portfolio overlap (e.g., the core capability of intelligence and information sharing overlaps with public information and warning). To avoid double counting, the formula above with the summations ∑ could not be used. The rollup method that was used instead is described in Appendix B.1. For a worked example of a detailed calculation of an NBCR using the BA model described in this section see Appendix B.2.

4.4 Data Sources HSSEDI obtained HSGP data in the BSIR, which is a comprehensive list of all HSGP grants between 2006 and 2015. Grants were issued in the following mission areas:

• Prevention

• Mitigation

• Protection

• Response

• Recovery The BA pilot used the Prevention mission area as an exclusive area of focus. These investments are intended to reduce the frequency of terrorist events (e.g., finding and disrupting terrorist cells, deterring attacks from initiating) and reducing the frequency with which attacks result in damage (e.g., by increasing timing alerts/warnings, decreasing breaches into secured areas, deterring attacks away from densely populated areas). These projects work together as a system to prevent and reduce the impacts of terrorism. The HSGP Prevention mission area shows investments made in five core national-level capabilities:

• Intelligence and Information Sharing

• Public Information and Warning

• Screening, Search, and Detection

• Interdiction and Disruption

• Forensics and Attribution

The number of projects was flexible but was limited by the objective to choose the smallest set that spanned all the key characteristics of each of the Prevention core capabilities. Table 14 lists the project samples.

68

Table 14. Prevention Core Capabilities and Sample Projects

Core Capability Reduction of Risk in Terrorism

Intelligence and Information Sharing

Enhance information and vulnerability analysis and information dissemination capabilities

Establish connection between the New Jersey (NJ) Information Sharing Environment (NJISE) and NJ Data Exchange (NJDEx)

Establish/enhance a terrorism intelligence/early warning system, center, or task force

Continue support of the New York State Intelligence Center (NYSIC)

Public Information and Warning

Establish/enhance citizen awareness of emergency preparedness, prevention, and response measures using Short Message Service (SMS) messaging

Establish/enhance citizen awareness of emergency preparedness by fielding outdoor warning sirens

Establish/enhance citizen awareness of emergency preparedness by fielding a long-range acoustic device

Establish/enhance a terrorism intelligence/early warning system

Screening, Search, and Detection

License plate readers

Explosive detection canine team

Personal protective equipment and hand-held explosive detector

Law enforcement flight operations

Interdiction and Disruption

Enhance explosive ordnance disposal units/bomb squad

Enhance capability to respond to all-hazards events

Develop homeland security/emergency management organization and structure

Enhance a terrorism intelligence/early warning task force

Forensics and Attribution

3-D imaging equipment for crime scene investigation

Enhance radiological detection capabilities

Enhance forensic analysis and attribution capabilities.

UASI and OPSG funded the Prevention core capability projects. UASI grants must be used in projects that enhance a city’s capacity to train, organize, equip, and prepare for terrorism. OPSG funds were allocated based on risk-based prioritization using a U.S. Customs and Border

69

Protection (CBP) sector-specific border risk methodology that include, but is not limited to, threat, vulnerability, miles of border, and other border-specific “law enforcement intelligence.” The BA approach focuses on using the GTD to construct a national-level estimate of the exposure of the United States to losses from terrorism, including fatalities, injuries, and property damage. The focus of this effort is to assess how effective the Prevention Portfolio investments were at reducing these losses or exposure to them. Reductions in losses were equated to benefits. Table 15 shows data from the GTD, which were used in the BA to compute the losses of a terrorist event. Avoided losses via prevention are the benefits of the Prevention Portfolio.

Table 15. GTD Provides a Basis to Compute the Uncertainty of Terrorism Losses

Data Min. Med. Max. Source/Notes

Frequency of Terror Events Per Year (FT)

6 28 61 GTD (extract of only U.S. events from GTD per year since 1995)

Fraction of Events that Cause Injury or Fatality (FFI)

0 22% 70% GTD (extract fraction of U.S. events causing fatality or injury per year since 1995)

Fraction of Events that Cause Property Damage (FPD)

33% 68% 93% GTD (extract fraction of U.S. events causing property damage per year since 1995)

Conditional Distribution of Fatalities (CDF)

1 1 1383/ 190K

GTD (extract data of fatalities from all events in the United States since 1995); FEMA “max of max” planning prescribes 190K

Conditional Distribution of Injuries (CDI)

1 1 7366/ 265K

GTD (extract data of injuries from U.S. events since 1995); FEMA “max of max” planning prescribes 265K

Value of Statistical Life (VSL)

$5.4M $9.6M $13.4M DOT Policy Guidance from 2016

Fractional VSL of Injury (VI)

0.03 0.11 0.593 DOT Policy Guidance from 2016 (as applied to Oklahoma City injuries after attack.)

Conditional Distribution of Property Damage (CDP)

$50 $86K $1.08B GTD (extract of global terrorism data for property damage events since 1995 due to insufficient U.S. data)

Conditional Monetized Value of Fatalities and Injuries (MFI)

$9.2M $27M $1.2T MFI = VSL * (CDF + CDI * VI)

Baseline Annual U.S. Exposure to Terrorism (BTR)

$175 $147M $3.6T BTR = FT * (FFI * MFI + FPD * CDP)

70

Data Min. Med. Max. Source/Notes

Baseline Annual U.S. Exposure to Terrorism (BTR): Property Damage

Mean $5.47B

The mean value between the Min and the Max of the Baseline Annual U.S. Exposure to Terrorism (BTR).

Baseline Annual U.S. Exposure to Terrorism (BTR): Fatalities/Injuries

Mean $1.48

The mean value between the Min and the Max of the Conditional Monetized Value of Fatalities and Injuries (MFI).

4.5 BA Results

4.5.1 SME Elicitation For the BA pilot, HSSEDI assembled a terrorism prevention SME panel with extensive experience in terrorism intelligence and physical security. Appendix B.3 provides the qualifications of the members of the HSSEDI terrorism prevention SME panel. HSSEDI also developed an instrument for SME elicitation. The elicitation instrument includes 19 investments representative of the Prevention Portfolio. Appendix B.4 identifies, describes, and summarizes the projects submitted to the terrorism prevention SME panel. The associated critical tasks in the right-hand column are the key characteristics of the core capability that the project satisfies. Figure 14 displays a project worksheet sample provided to the SMEs. The project information was formatted to facilitate the SME assessment.

71

Figure 14. Sample SME Elicitation Page

Before the elicitation session, HSSEDI provided the SMEs with uncertainty calibration training. The goal of calibration was to enable the SMEs to provide probabilistic ranges that include the actual probability at least 70 percent of the time.

4.6 SME Calibration Training and Elicitation The group had a calibration score of 38 percent after the first round of calibration.47 The elicitation session began with an overview discussion of the elicitation goals and methodology. The SMEs engaged in conversation and collaborated to clarify and validate their understanding of the project descriptions. In some cases, a SME had more knowledge than the rest of the group in an area such as nuclear radiation detection. This pseudo-behavioral approach deviates from the classical methodology in elicitation because the mathematical approach typically does not permit interaction between experts. This pilot study combined the behavioral and mathematical approaches in response to requests from the subject matter experts. As a result,

47 The purpose of the calibration scores is to be able to determine the average calibration score for the group of SMEs who will be performing the project expert elicitation. The score tells us what degree of confidence to place on the project scores. For example, if a group of SMEs had a calibration score of 38 percent that means that 38 percent of the time, we can expect their average project elicitation answers in the form of an interval (low to high percentages) to contain the true value of the project’s effects on reducing terrorism losses.

72

this limits the interpretation of the SME results as truly independent since the objectivity of each SME’s response could have been influenced during these conversations. After the elicitation session was over, the results from the calibrated48 trained SMEs were compared with those from the one untrained SME to determine whether the calibration training made a difference. Students’ t-tests were conducted to determine if the calibrated SMEs provided percent reduction estimates that were statistically different from those provided by the uncalibrated SME. Three students’ t-tests were conducted: one for lower bound estimates, one for most likely estimates, and one for upper bound estimates. All three of the t-tests showed that the uncalibrated SME’s estimates were, in fact, statistically different from those provided by the calibrated SMEs. Based on these results from the t-tests, it is concluded that the calibration training does make a difference. The decision to allow an uncalibrated SME to participate in the elicitation session was a tradeoff decision. HSSEDI deemed it more important to have the terrorism prevention expertise on the panel be included along with the expertise from the other SMEs, than to not allow the SME to participate at all because he hadn't been calibrated. HSSEDI conducted a statistical test of whether there was a statistically significant difference in the scoring of the uncalibrated SME as compared to that of the calibrated SMEs. We found that there was. Two lessons have been learned from that finding. First, calibration does make a difference in how SMEs express their uncertainty. Second, uncalibrated SMEs should not be allowed to participate in future elicitation sessions.

4.6.1 SME Assessment of the Project Investments Four SMEs were asked to estimate the effects of 19 projects across five core capabilities. For each project the SMEs provided six estimates: the project’s minimum, most likely, and maximum reduction of the probability of successful attack and the project’s minimum, most likely, and maximum reduction of the probability of impact given a successful attack. The five core capabilities provided the total values for the Prevention Portfolio. The SMEs recorded their estimates in identical paper forms that consisted of one page per project. Each page included the core capability the project is associated with, a description of the project, and an area for the SMEs to record the percent reduction in 1) terrorist attack frequency and 2) likelihood of impacts for each project. Each SME recorded low, most likely, and high attack probability reductions and low, most likely, and high impact probability reductions in percentages. The low, medium, and high percentages were then averaged across all SMEs for each project. Next, the average results for the projects associated with a single core capability were calculated. That calculation provided for the core capability the low, most likely, and high estimates of the

48 The calibration scores were calculated by counting the number of correct answers the SME gave when expressing an uncertainty interval that was supposed to bracket the answer that was true but unknown to him/her. The number of correct answers was divided by the total number of questions to obtain an individual SME score (grade) for each iteration. The individual SME scores were averaged for each iteration.

73

percent reduction in attack frequency and low, most likely, and high estimates of the percent reduction in the likelihood of impacts. The details of the SME-provided estimates, the average project estimates, and the core capability estimates for each of the five core capabilities are in Appendix B-4. Table 16 presents the results of combining the five core capabilities into values for the Prevention Portfolio in accordance with the algorithm described in Appendix B.1.

Table 16. Total Prevention Portfolio, Combination of the Five Core Capabilities

Upper Bound Lower Bound 50% Estimate Attack

Low 0.32 0.12 0.22 Most Likely 0.64 0.30 0.47 High 0.84 0.43 0.63

Impact Low 0.37 0.11 0.24 Most Likely 0.66 0.24 0.45 High 0.86 0.37 0.61

4.6.2 Confidence in SME Results As previously described, during the elicitation session, the SMEs provided an uncertainty range for the reduction probability by estimating a lower and upper bound. There were two uncertainty ranges for each project: one for the reduction in the probability of attack and the other for the probability of reduction of impact. The average uncertainty range was .23. These uncertainty ranges demonstrate the confidence that SMEs had in their own estimates, yielding a higher confidence that HSSEDI attributes to their results as well.

4.6.3 Graphing and Tabulating the Net Benefit Cost Ratios Based on the SME outputs and rollups above, NBCR ranges were generated from the BA model, graphed, and tabulated. The graphs show a box encompassing a region inside the space of all possibilities (1 percent – 100 percent attack frequency reductions; 1 percent – 100 percent impact reductions) that represent the average49 of the SMEs’ assessments of the projects within that core capability. The most extreme range of values are the low-low and high-high values for the reductions in attack frequency and impact, respectively. The most likely value is also plotted

49 Using median, as opposed to mean of the impacts (both property damage loss and deaths/injuries), would have had a very large effect on the results. To illustrate this large effect, let’s take “most likely” value for the Prevention Portfolio as a whole. The mean values that were used to calculate the value in the cell for the “most likely” Prevention Portfolio value of 2.56 net benefit cost ratio (as well as all other values in the spreadsheet BA model) was $ 5.47 billion for Property Damage and $1.47 billion for deaths/injuries. If instead the median values of $147 million for property damage and $27 million for deaths/injuries were used, the resulting NBCR would have been - 0.92 (negative .92). The net benefits would not have been greater than the costs.

74

within the boxed region as the average of the most likely values of the projects according to the SMEs. For the analysis, NBCR ranges were generated from the BA model, graphed, and tabulated. Figure 15 shows the sum of the five ranges, Table 17 shows the cost ratio ranges, and Appendix B.9 shows the detail of each of the five core capabilities that compose the Protection Portfolio.

Figure 15. Mapping of Overall Prevention Portfolio Net Benefit Cost Ratio Range

75

Table 17 summarizes the NBCR ranges for all core capabilities and for the Prevention Portfolio.

Table 17. Summary of Net Benefit Cost Ratio Ranges

Core Capabilities and Prevention Portfolio

Net Benefit Cost Ratio Ranges* (Low to High)

Intelligence and Information Sharing − 0.6** to 3.24

Interdiction and Disruption − 0.48 to 5.54

Screening, Search, and Detection; 0.59 to 14.47

Public Information and Warning 0.9 to 16.1

Forensics and Attribution 53 to 341

Overall Prevention Portfolio − 0.13 to 4.82

Most Likely: 2.56

*Based on SME calibration, there is a 38 percent chance that true value lies within these uncertainty ranges.

**Negative numbers are produced as a logical result of the fact that the net benefit cost ratio is defined as (B-C)/C. Therefore, for smaller percentage reductions (i.e., smaller benefits), the benefits will be less than the cost and (B-C) will be negative. Hence, there is almost always a red (negative) region in the colorized mappings for the Core Capabilities.

4.7 Findings The BA is an exploratory pilot/feasibility study and does not produce findings that prove causality. The results should be interpreted with caution given the nature of the methodology. Table 18 shows the results at the national Prevention Portfolio level.

Table 18. BA Results

Net Benefit Cost Ratio Range -0.13 to 4.82

“Most Likely” Value 2.56

Interpretation of “Most Likely” Value Each $1.00 of prevention investment yields $3.56 of benefit

The overall findings from the BA were: • The Prevention Portfolio has a net benefit cost ratio range of − 0.13 to 4.82 with a “Most

Likely” NBCR of 2.56 within that range. • The Prevention Portfolio annual dollar benefits range from $190 million to $1,269

million compared to a mean of $218 million for FEMA Prevention investments (2010 – 2015)

76

• The “most likely” benefit is $776 million. Hence, each $1.00 of Prevention Portfolio investment yields $3.56 of benefit.

• The Prevention “most likely value” pays for 91 percent of the total HSGP investment (3.56 x $218 million = $776 million compared to $854 million for entire HSGP).

The following issues and observations should be considered when planning to apply the BA to additional HSGP portfolios.

• The calculation of the NBCR requires better explanation and guidance to be properly interpreted by the SMEs in making their project level assessments. Some SMEs were bothered the fact that the multiplier L in the equation (L represents national annual U.S. losses from all forms of terrorism) is a national-level cost from all terrorist acts rather than just the cost from the terrorist acts associated with each core capability category. The session facilitator will make this point explicit in his/her guidance in further iterations and applications of the course of action so that SMEs can take it into account in their attack and impact reduction assessments.

• There appears to be a systemic budget bias in the NBCR for the core capabilities: as one moves from higher funded core capabilities to lower funded core capabilities the NBCR systemically increases. This systemic bias could be offset in future applications of the method by revealing to the panel of Terrorism Prevention SMEs the average budgets for each of the core capabilities prior to their making estimates of the effectiveness of projects within them.

• The assessment of individual projects as a national-level capability (supporting a national core capability) was a challenge for the SMEs. Yet, this is what makes the methodology scalable. By not having to assess every project under a core capability but rather only a small sample of representative projects beneath it, one can calculate NBCRs for a small, manageable set of core capabilities and their rollup to the Portfolio (Mission Area) level.

4.8 Recommendations HSSEDI has the following recommendations based on the BA.

• Identify the other mission area portfolios that could be assessed with the BA method. Of the five mission area portfolios in all—Prevention, Protection, Mitigation, Response, and Recovery—the first three (Prevention, Protection, Mitigation) could be assessed with the BA method. Protection and Mitigation have a terrorism deterrence effect associated with them in addition to their primary impact reduction effect. Hence, those portfolios could have their projects assessed for both reduction in terrorist attack frequency (due to deterrence) and reduction in impacts. Response and Recovery do not have a deterrence aspect to them and hence cannot be assessed with the BA approach. Increased HSGP coverage is shown, for example, by the fact that Protection plus Prevention combined account for, on average, 60 percent of the 2010-2015 HSGP budgets.

• Pursue external peer review and independent validation of the BA methodology.

• HSSEDI foresees improving the accuracy of its SMEs for the Prevention Portfolio, as well as the Protection and Mitigation Portfolios, by having them trained/calibrated by the

77

firm Hubbard Decision Research using the company’s three-hour training webinar, “Calibration – How to Quantify Your Uncertainty” (https://www.hubbardresearch.com/). Each webinar student takes a series of calibration exams during the webinar and is given a final score after the last one. The final scores of all the students can be averaged to determine the overall calibration score for any SME cohort group.

• In the event the BA model is put into production use by FEMA, HSSEDI recommends using a panel of nationally recognized terrorism experts for the elicitation.

• Historical data on the frequency of attacks could be used in future applications of the model. For example, from the GTD, one can calculate the mean of the number of terror attacks in the United States from 1995 – 2016: 28 annually. Hence, on any given day within the United States there is 7.7 percent (28/365) chance of an attack. The 7.7 percent could be used to replace the first factor in the formulation of the model, the PT (the probability of a terrorist attack). Using this value in future applications of the BA model will relieve the SMEs of the burden of having to estimate it as part of their estimation of the combined factors (PT * PH|T) during the elicitation session.

• In the longer term, assess whether SMEs can ultimately be replaced by improved data and simulation models in future applications of the BA. o Improved data could include, for example, the mapping of each project to its THIRA

target. It could also include the amount of quantitative capability improvement in its application area that the project provides. Having the latter would allow the possibility that SMEs can be replaced by computer simulation models.

o How available and scalable are simulation models for the types of HSGP projects that represent core capabilities?

o Can those models simulate reductions in probability of attack and likelihood of impacts resulting from FEMA HSGP projects?50

50 The Terrorism SME Expert System could be developed as follows. HSSEDI would establish a project team from personnel in its Cognitive Science and Artificial Intelligence technical center. The team would identify nationally recognized terrorism SMEs. The terrorism SMEs would be convened, and the development team would elicit expert judgements from them. Those judgements would consist of their estimates of the amount of reduction in the probability of attack and the amount of reduction in its impacts that each sampled HSGP project would provide. Projects would be sampled from the Prevention, Protection, and Mitigation Portfolios. Those SME judgements, together with the technical description and characteristics of the HSGP projects and other historical information (e.g., from the GTD), would be fed into the Terrorism SME Expert System by the development team.

https://www.hubbardresearch.com/

78

5 Next Steps When considering how to advance these pilots to the production phase, HSSEDI considered the following:

• Ready availability of data o Are the data available or easily collectable? o How much data mining and manipulation are required to expand the scope of the pilot

to production?

• Ease of scalability o Did the pilot model reveal a design that can process a significant increase in the

amount of data without significant redesign or accommodation?

• Revisions required to the pilot studies to move them to production o What structural changes are required to the model to expand the area of examination

(use of more data, locations, number of years, types of grant-funded projects, etc.)

5.1 Transition to Production Assessments The DOM pilot revealed complications and issues related to ambiguity and cross-correlation in the targets, core capabilities, and investment descriptions. The degree to which investments contribute to the achievement of targets was not clear. Though HSSEDI attempted to construct relationships between investments, capabilities, targets, and observations of performance, substantial work continues to be needed to fuse the efforts into a cohesive and unambiguous network or system. The cross correlation between capabilities and targets that has not been addressed in the lexical use of the categories will obscure any attempt to apply a disciplined approach to quantifying these relationships. These relationships need to be standardized in order to provide clarity in the development of projects/investments and clear direction in the estimation and measurement of performance. The DOM pilot was run with data from one state, one year, and one core capability. To expand the scale of this pilot will require addressing all the above issues before scaling. Given the complexity of the model itself, it is not unreasonable to expect additional adjustments to the methodology will be revealed when expanding the pilot. Prior to conducting the RPA pilot as a formal analysis, all states and urban areas should be canvassed to determine which ones allocate HSGP funds through a formal process like Virginia’s. Virginia was selected for conducting this pilot because the data were available, but the prevalence of their formal vetting and allocation process is unknown. If a relatively low portion of HSGP funds are competitively awarded, then expanding the pilot would create cost-benefit ratios that are relatively uninformative of the overall HSGP program. On the other hand, if data exist for a sizeable portion of HSGP awards that can be used to infer SDM value from a formal vetting process, then expanding this pilot could provide meaningful insight. The project review and funding decision-making process used by states is not standardized. As a result, the data collected by the states for that process may vary significantly from state to state. Given the uncertainty of the data available and the format they are in, the time and level of effort required for collecting and standardizing data from a statistically significant sample of states is unknown.

79

One statistical issue that should be explored if this pilot is converted to production is the assumption that funding requests are exogenous to the funding decision. If applicants lower their funding request to increase the chances of being selected, then the funding request may be creating endogeneity bias in the demand specification. If so, then an identification strategy, such as instrumentation, would be necessary. The BA pilot utilized the GTD—the data were easily available, standardized, and easy to manipulate. Furthermore, this database can be used to expand the pilot or convert it to a full study for the Preparedness, Protection, and Mitigation Portfolios with no need to seek additional data. The model itself would require little if any modification to expand the scope. The most significant change to the BA to move it into production relates to the SMEs. Standardized training for all the SMEs and additional rounds of elicitation will be required to improve the confidence levels of the results. Both are easily accomplished at a modest cost.

5.2 Recommendation HSSEDI recommends transitioning the BA from a pilot study to full production by redoing the preparedness portfolio and expanding the study to include the Protection and Mitigation Portfolios. All this should occur after the SMEs have undergone calibration training provided by a calibration specialist. The elicitation process should include four rounds and only SMEs who have undergone that training. The results of this work should provide FEMA NPAD with a BA from which they can determine if the benefits of the HSGP grants exceed the costs.

80

Appendix A: DOM A.1 Definition of Benefit Categories This table describes the similarities between the DOM pilot benefit categories and examples found in literature. The DOM pilot benefits are a high-level synthesis of the literature examples.

Table 19. Comparison of Benefit Categories

DOM Pilot Benefit Categories

Benefit-Cost Analysis of FEMA Hazard Mitigation Grants

Rose et al.

Cost-Benefit Analysis of Disaster Risk Reduction

Aktion Deutschland Hilft e.V. Reduced risk of lives lost Reduced societal losses deaths, injuries, and

homelessness Intangible and Direct: Loss of life, injuries Reduced risk of

injuries/illnesses Reduced risk of property damage

Reduced direct property damage; e.g., buildings, contents, bridges, pipelines

Tangible and Direct: Structural damages

Reduced risk of business losses

Reduced direct business interruption loss; e.g., factory shutdown from direct damage Reduced indirect business interruption loss; e.g., ordinary economic “ripple” effects

Tangible and Direct: Structural damages, inventory loss, loss of agricultural land Tangible and Indirect: Production downtime, business disruption

Reduced risk of time spent without necessities

Lifeline interruption

Reduced risk of impact on other QOL issues

Reduced nonmarket environmental damage; e.g., wetlands, parks, wildlife; Reduced other nonmarket damage; e.g., historic sites

Intangible and Direct: Destruction of cultural heritage Intangible and Indirect: Increased vulnerability, loss of confidence, migration, disruption of school attendance

Reduced emergency response; e.g., ambulance service, fire protection

Note: Benefits are equated to reduced or avoided damages as described in the table.

81

A.2 SAR Subject Matter Experts Table 20 describes the qualifications of the three Coast Guard SAR SMEs who assessed the contributions of THIRA targets to benefit categories.

Table 20. SME Qualifications

Name Title Qualifications DOM SME #1 Project Leader

MITRE Homeland Security Center

Interagency Coordinator, DEEPWATER HORIZON oil spill response. U.S. Coast Guard first responder, 1998-2006: Chief of Marine Environmental Response, Chief of Port Safety and Security, and Chief of Planning and Risk Management – Hampton Roads Captain of the Port Zone; Chief of Port Operations, Chief of Response and Planning – Savannah Captain of the Port Zone. Master’s Degree – Homeland Security and Defense. Trained in Marine Firefighting and Oil and HAZMAT Response. ICS Type II Liaison Officer. USCG Explosive Handling Supervisor.

DOM SME #2 Senior Principal, Systems Planning and Analysis

Assigned to DHS Security staff leading federal inter-agency working group to enhance incident planning and response across the USG and overseeing preparations for national level exercises. Twenty-four years’ experience in U.S. Coast Guard Search and Rescue and response operations.

DOM SME #3 System Engineer MITRE Homeland Security Center, Commander, U.S. Coast Guard (Ret.)

23 Years of Search and Rescue experience on eight Cutters and five Navy ships. Principle author of the JP 3-50 and 3-50.1 National SAR Manuals Vols. 1 and 2. Led maritime contingent in several ICS consequence management actions around the United States. Participant in Joint DOD/DHS/SLTT planning/training/ exercises/responses for emergency situations.

82

A.3 Calculation of Investment Benefit Table 21 illustrates the calculation of the benefit of a single SAR investment to the jurisdiction. The fractional benefit is calculated with the target score of 7.5 as the percent change in the number of rescues. (This is the output of the systems dynamic simulation for the Tallahassee SAR Investment 5.)

Table 21. Calculation of Individual Investment Benefit

Blank cells represent zero values. Note that the score shown does not necessarily represent the total benefit of the example investment since investments may have benefits that extend beyond the search and rescue scenario that was considered in the pilot. Benefit category scores – Calculated as the normalized weighted average of the target scores for that category. Benefit category weights – HSSEDI SME assessments.

83

Investment benefit score – The overall benefit of the selected investment is the normalized weighted average of the benefit category scores. This is the benefit score for Tallahassee Investment 5. Fractional benefit – The product of the investment benefit and the jurisdiction’s fraction of the state population. For this example, Investment 5 (Tallahassee), the fractional population is 0.43 percent and the fractional benefit is 0.0015.

84

A.4 Simulation Inputs and Sources Table 22. Simulation Inputs, Sources, and Descriptions

Ingest Field Name Source Description Id HSSEDI Unique identifier assigned to each record legal_agency_name BSIR Jurisdiction linked to each investment investment_number BSIR Investment number investment_name BSIR Name of investment core_capability_mapping BSIR User-populated core capability Subcategory BSIR User-populated subcategory project_title BSIR User-populated project title project_type BSIR User-populated project type project_description BSIR User free text field describing the investment funding_type BSIR Type of funding funding_amount BSIR User-populated funding amount requested Notes HSSEDI Annotations used when reviewing investments for group

assignment Grouping HSSEDI Assigned categorized grouping based on review of projects in

each investment Pop U.S. Census51 Demographic data by jurisdiction pop_fem HIFLD52 Fire and EMS station data derived from DHS’s Homeland

Infrastructure Foundation-Level Data (HIFLD) for fire stations and EMS stations, combined and calculated as a per 100,000 residents value

rescue_rate HSSEDI Derived using residents and pop_fem rescue_adj HSSEDI Asserted adjustment to rescue rate

51 American Community Survey (ACS), 2011-2015, U.S. Census Bureau, https://www.census.gov/geo/maps-data/data/tiger-data.html 52 Homeland Infrastructure Foundation-Level Data (HIFLD), U.S. Department of Homeland Security, https://hifld-geoplatform.opendata.arcgis.com/

https://www.census.gov/geo/maps-data/data/tiger-data.html


https://hifld-geoplatform.opendata.arcgis.com/


85

A.5 Data SourcesThe data sources used for the DOM pilot, and how they were applied, are listed in Table 23.

Table 23. Data Sources Reviewed

Name/Title Information Reviewed Use BSIR Source Data 2006-2016

Jurisdiction, investment description, core capability alignment, project description, funding, other

Inputs to MAUT and Sim models

State Preparedness Report (SPR)

Priority ratings(H/M/L) by capability, limited free text gap data, standard gap categories

MAUT value tree structure

THIRA Threat/hazard by state, targets/scenarios, resource requirements


FEMA Case Studies (DC, Maine, Montana, Ohio)

Grant proposal evaluation criteria and weights, projects aligned to core capabilities of interest (priorities and gaps) and their outcomes


National Preparedness Goal FEMA Preparedness Goals/Objectives/Practices MAUT value tree structure National Preparedness Report

National level preparedness gaps, core capability priorities, major events, results


EMPG (Emergency Managment Performance Grant) ROI Report NEMA IAAEM 2017 Final. PDF

Value of plans and resources provided through EMPG grants that enabled event preparedness. The information was for EMPG grants, but the value attributions to investments may be transferable to SHSP/UASI.


After-Action Reports For Events: location, loss of life, damage details and cost, issues/gaps, improvements needed. For Exercises: Description of exercises some including goal of exercise, few with performance results.

Reviewed but not used for Sim and MAUT due to lack of effects-related data and inconsistent for use in deriving rescue rates

FEMA Risk Index 20161128 - short. Pptx

Level/rank/indicator of risks. Useful for regional project prioritization, resource allocation, mitigation/land use planning.

Inputs to Sim and MAUT for jurisdictional data

CBA/ROI studies Anecdotal reports of effects/value of planning, exercises, and speed of response for potential support of MAUT weighting assumptions


VDEM data Methodology used by Virginia Department of Emergency Management for prioritizing competitive investments; data from that process for 2013-2017


USCG SAR data SAR case data for 2017 Reviewed but not used for Sim and MAUT due to it its lack of ability to represent the jurisdiction-specific rescue rates

86

Name/Title Information Reviewed Use State Hazard Mitigation Plans

State-level hazard probabilities Not suitable for use

87

A.6 Data Findings The 2004-2016 HSGP investment data are characterized by weaknesses in four major categories:

• BSIR data lacks descriptions of linkage(s) between investment and benefit

• No defined benefit measures (e.g., reduced risk of lives lost)

• Lack of preparedness capability baseline

• Inconsistent, non-standardized investment descriptions

A.6.1 Data Improvement Recommendations Over the course of developing the DOM pilot, HSSEDI made numerous observations regarding data improvements. This section describes recommended changes to HSGP data. 1. Link BSIR data to THIRA targets Create a new identifier field connecting each BSIR record to the THIRA target(s) that it supports. Create a new field that allows grantees to assess the degree to which the project helps achieve the target. Reliable assessments of the degree of contribution rest with the grantee and their knowledge of the jurisdiction’s preparedness needs. The expectation is that proposed projects would not be scored with a utility of ‘0’. However, this same process could be used by SLTTs to vet their proposed projects prior to submission to FEMA; in this case, SMEs/stakeholders may assess a project with a “0.” HSSEDI suggests using a scale like the one below. An ordinal scale was selected since, in this case, it is the order of the values (high, medium, or low impact) that is important and significant. The differences between each one are not really known (i.e., we can’t precisely measure the difference between a project’s medium or high impact). The number of points in the scale is determined by the number of separate, meaningful assessments that can be made. To simplify the SME scoring exercise, assign a distinct utility score on a 0-100 scale for each impact with equal separation (40 points) between each impact level.

Table 24. Utility Score Descriptions

Impact Description Utility Score

High The project makes a significant contribution to meeting the target. 90 Medium The project makes a moderate contribution to meeting the target. 50 Low The project makes a minimal contribution to meeting the target. 10 None The project does not contribute to meeting the target. 0

Anticipated Benefit Linking projects to the targets defined in the THIRA will allow FEMA to determine which funding is being directed towards which specific investment outcomes or benefit categories.

88

Knowing the degree to which the project will help achieve the target provides a metric that can be tracked to assess the benefit the project provides. 2. Include Investment/Project Benefit Areas in BSIR Include the county-level (or equivalent) FIPS code for any geography that the investment or project will serve. For example, a training exercise may benefit multiple jurisdictions included in a mutual aid agreement. Projects affecting specific organizations/sites are rolled up to the county level. FIPS codes are the most common geographical identifier found in hazard-related geospatial datasets such as HIFLD. Anticipated Benefit Including specific benefit location information will improve the understanding of the investment or project impact, including the impacted population. This can be especially helpful when scaling the simulation component of the DOM. 3. Standardize State Hazard Mitigation Plan Content Standardize the format of the SHMP, particularly with concern to the hazard identification and risk assessment. To the extent possible, SHMPs should draw from a standardized list of hazards and the elements of the risk assessment; for example, the probability of a hazard should be described consistently across hazards and SHMPs. It is recommended that the hazard data are provided at both the state and county-equivalent level. Anticipated Benefit Standardizing SHMPs ensures that consistent data for the population at risk for all relevant hazards at the state and county level are provided for HSGP decision makers. This facilitates the comparison of hazards and impacted population between SHMPs and, by extension, the benefits of investments or projects. Additionally, improved population-at-risk input data will ease the scalability of the DOM simulation model (e.g., state-level hazards and county-level populations at risk aligned to specific hazards). 4. Define baseline capabilities against targets Baseline preparedness capabilities against THIRA targets should be included in investment and project descriptions. For example, assuming targets are the measure, each target would need to be reviewed to decompose each one to the rates that incrementally would change the target values. Each of these rates would then need to be mapped to either data to derived rates or mapped to investments in order to assess how much each would potentially impact the rates and therefore target values. With the SAR example, HSSEDI decomposed the “Conduct search and rescue operations” and its reduction in lives lost into the rate at which the impacted population is rescued. “Evacuate people following notice of impending event” would probably impact all the benefits. The rates that would matter for this target might Apply to, for example, early warning and detection technologies, communication, and coordination training for improved reach and efficiencies. Anticipated Benefit Establishing capability baselines allows the measurement of incremental preparedness capabilities against targets and thereby the benefit of investments or projects.

89

5. Standardize domain values Standardize the way information is collected. Provide a standard set of domain values from which the user must select and minimize the use of free text responses. Anticipated Benefit Standardization of collected data ensures data quality and facilitates the ability to do data analysis with minimized data cleaning efforts in the future.

90

Appendix B: Breakeven Analysis B.1 New Algorithm for Combing Core Capabilities at the Prevention Portfolio Level This appendix describes how the percent reduction in the attack and impact (low, most likely, and high) probabilities for the five core capabilities were combined to produce the percent reduction in the attack and impact (low, most likely, and high) probabilities for the Prevention Portfolio. This appendix describes the new algorithm by example. It describes how one of these Prevention Portfolio reductions in probability is calculated, but the calculations for all six Prevention Portfolio reductions (attack and impact [low, most likely, and high]) are identical. After the reduction probabilities have been calculated for the five Prevention core capabilities, they must be combined to produce a reduction probability for the total Prevention Portfolio. The problem is that we do not know the amount of dependency there is between the core capabilities. That is, are they reducing the probability by mitigating unique threats or the same threats using unique techniques or duplicative techniques? Because these interdependencies cannot be quantified without evaluating all projects, HSSEDI estimated them as the average between the lower bound that assumes completely dependent projects and the upper bound that assumes completely independent projects. The lower bound assumes that core capabilities have complete dependencies, meaning they are reducing the attack or impact probability by mitigating the same threats. The lower bound situation is depicted in Figure 16, where the circle depicts the attack or impact probability being reduced and the pie slices represent the reduction in the probability resulting from the core capabilities. For the lower bound, the percent reduction is calculated as the largest percent reduction of the core capabilities, e.g. the pie area showing the reduction in probability from core capability one.

Figure 16. Lower Bound on the Probability Reduction

91

The upper bound assumes that core capabilities are completely independent, meaning that they are reducing the attack or impact probability by mitigating different threats using different techniques. The upper bound situation is depicted in Figure 17, where the circle depicts the attack or impact probability and the pie slices represent the reduction in the probability resulting from the core capabilities.

Figure 17. Upper Bound on the Probability Reduction

For the upper bound, the percent reduction is calculated as follows:

Given:

R(i) is the probability reduction for Core Capability i, i = 1 to 5.

Calculate:

R = 1 – ((1-R(1))* (1-R(2))* (1-R(3))* (1-R(4))* (1-R(5))) where R is the upper bound on the probability reduction for the total Prevention Portfolio.

The unknown actual Prevention Portfolio probability reduction value is assumed to be the value at the midpoint between the lower and upper bounds of the combined core capabilities and is calculated as the mean of the two bounds (lower and upper). This value represents an assumed 50 percent dependency (overlap) among the Prevention Portfolio core capabilities.

92

B.2 Sample Detailed Calculation of a Prevention Portfolio Net Benefit Cost Ratio Overall formula for Net Benefit Cost Ratio: (Benefit – Cost)/Cost Example for Prevention Portfolio “most likely” value = 0.54 See Figure 15, Mapping of Overall Prevention Portfolio Net Benefit Cost Ratio Range, at the cell where the 30% reduction in Impact and 30% reduction in attack frequency intersect: Value = 0.54. Formula 1: ($C14*(J$6*AnnualRisk!$V$27+J$7*AnnualRisk!$S$37) − AB1)/AB1 = 0.54 Variable values from right to left () in above formula: AB1 = $204428,714 (cost of Prevention Portfolio investment) [Tab for Prevention Portfolio] AnnualRisk!$S$37 = $5,473,298,906 (mean Baseline U.S. annual exposure to terrorism) [AnnualRisk Tab. Cell: S37] J$7 = 0.118 (30% reduction in Impact of property damage) [breakevenROI Tab. Cell: J7] AnnualRisk!$V$27 = $1,481,839,818 (monetized fatalities and injuries: mean value) [TabAnnualRisk. Cell: V27] J$6 = 0.3199 (30% reduction in Impact of fatalities and injuries) [breakevenROI Tab. Cell: J6] $C14 = .28 (30% reduction in frequency of terrorist attack) [breakevenROI Tab. Cell: C14] Filling in the above values in Formula 1. for Prevention Portfolio “most likely” value = 0.54, we have:

93

= (.28*(.3199*1,481,839,818 + .118*$5,473,298,906) − 204,428,714)/ 204428,714 = (.28*(474,040,557 + 1119889829) − 204,428,714)/ 204428,714 = (313569152 − 204,428,714)/ 204,428,714 = (109,140,438)/204,428,714 = 0.5339 The above value is more accurate than the value of .54 calculated in the BA spreadsheet.

94

B.3 BA Subject Matter Experts Tables 25 provide the qualifications of the members of the HSSEDI terrorism prevention SME panels.

Table 25. BA SME Panels

SME Qualifications BA SME #1 Supported the establishment of an intelligence organization tasked with a terrorism

prevention mission BA SME #2 Expert on countering terrorist weaponization of science and technology BA SME #3 Strategic planner and assessment analyst for the National Counter Terrorism

Implementation Plan BA SME #4 Counterterrorism analyst; social network analyst; intelligence analyst BA SME #5 Multi-discipline Systems Engineer and intelligence analyst; developer of Indications

and Warnings for Countering Terrorism – Chronoskope BA SME #6 MITRE DHS Prevention Portfolio Manager BA SME #7 Counterterrorism analyst; social network analyst; intelligence analyst BA SME #8 Since 2007, worked on Homeland Security projects, most of which were related to

reducing the risk and impact of terrorist attacks

95

B.4 Project Samples Table 26. FEMA Prevention Portfolio Core Capability – Project Samples

BSIR ID Project Short Name Project Description Critical Task

6114 Enhance information and vulnerability analysis and information dissemination capabilities.

This project enhances information analysis and law enforcement capabilities. It upgrades computers and software and training to more effectively gather and share information. For example, the project includes a License Plate Reader (LPR) for identifying possible threats, Los Angeles Regional Common Operational Picture Program to electronically gather and transmit data to stakeholders off-site, Video Downlink for digital high-definition video link systems serving multiple agencies, and Palantir equipment upgrade from various agencies to expand capabilities for better intelligence analysis and information sharing. This project supports Law Enforcement Patrol Team and Law Enforcement Aviation-Helicopters-Patrol and Surveillance National Incident Management System Typed Resource as well as other state and local typed resources.

1, 2, 5

6173 Establish connection between New Jersey Information Sharing Environment (NJISE) and NJ Data Exchange (NJDEx)

This project will allow for the connection of the NJDEx system to the NJISE. The NJISE enables the NJ law enforcement community to search and share police reports with participating police departments statewide. NJDEx provides critical support to the state’s investigative and analytical efforts as well. Additionally, NJDEx shares information with the FBI’s National Data Exchange (NDEx) program. Incorporating the NJDEx system into the NJISE environment provides a solution that allows for data interoperability services, bundled searches, and network services. The network services include NJDEX, HSIN, LEO, RISS, and others, as well as application services over time as part of an iterative process.

2, 5

6216 Establish/enhance a terrorism intelligence/early warning system, center, or task force

This project will maintain a statewide Suspicious Activity Reporting network for a holistic view of terrorism and criminal-related suspicious activity in Texas. This network facilitates comprehensive collection, analysis, and response to terrorism and crime-related activity between identified fusion centers in Texas participating in the network. This network will assist federal, state, tribal, and local law enforcement agencies in their efforts to detect, deter, and eliminate criminal activities. The Texas Joint Crime Information Center (TX JCIC) is utilizing TripWire Threat Detection and Analysis System for the state’s internal suspicious activity program.

1, 2, 3, 4

96


5931 Continue support of the New York State Intelligence Center (NYSIC)

The New York State Police (NYSP) will utilize funds to support the NYSIC, which is the Governor's designated Fusion Center for New York. The NYSIC will leverage grant funding to support the Phase 2 development of its Criminal Intelligence Analysis System (CIAS) for Intelligence Analysts, to support the Field Intelligence Officer (FIO) Program (to engage local law enforcement), and for supporting planning, training, and travel costs. These efforts will support all four COCs (Receive, Analyze, Disseminate, Gather) and the EC of Communications and Outreach. Also, the projects will sustain strengths in the NYSIC’s Fusion Center Assessment, as well as addressing gaps related to analyst training.

6, 7

Table 27. Interdiction and Disruption


6404 Enhance explosive ordnance disposal units/bomb squad

Dekalb County Police Department will use Fiscal Year (FY) 2015 HSGP funding in the amount of $127,014 to sustain and maintain current Explosive Ordnance Disposal (EOD) program. Funding will be used to purchase needed equipment and EOD/IED training devices for terrorism prevention and to sustain existing bomb disposal unit teams through the repair and replacement of worn equipment, upgrades to current equipment, and purchasing of new equipment as technology advances.

4

6472 Enhance capability to respond to all-hazards events

Glendale Police Department Special Operations Division responds to all chemical, biological, radiological, nuclear, and explosives (CBRNE) and Tactical calls for service within the city limits as well as neighboring jurisdictions when called upon. The division currently has no specialized vehicle in which to work command elements in the event of a CBRNE/Tactical situation involving extended crisis negotiations or communications support. The division has the necessary equipment, personnel, and training for these events, but lacks a specialized vehicle in which to contain and work. This project would entail the building and outfitting of a specialized vehicle to be used in these situations, to enhance the capability that already exists. As a city that hosts very large, high-profile events, the necessity of such a vehicle that is maneuverable has become very apparent.

1, 4, 5, 6, 7, 8,

9

97


6547 Develop homeland security/emergency management organization and structure

Region 6 plans to equip, train, and exercise response personnel and specialized teams to effectively respond to violent extremism and other hazards/incidents/Search and Rescue Operations. Funds will be utilized to sustain, maintain, and enhance local and regional CBRNE and All Hazard Response Capabilities, specialized teams, specialized support teams, operational interoperability, and operational coordination of capabilities and resources. This project will also provide funding to address gaps in Mass Search and Rescue Operations. On-Scene Security and Protection activities will address gaps in planning, equipment, training, and exercises to protect response personnel; and to secure disaster areas and law enforcement. Multiple agencies, jurisdictions and disciplines may submit projects under this investment.

1, 5, 7, 8, 9

6745 Enhance a terrorism intelligence/early warning task force

The Texas Rangers Special Operations Group (TRSOG) requires the capability to integrate state law enforcement all-terrain vehicle activities with federal partners to respond to, track, and trail criminals or lost subjects along the Texas-Mexico border. This capability will allow TRSOG to provide accurate distances from observation posts to ground, mobile and law enforcement air elements to enable the disruption, deterrence, and denial of transnational criminal smuggling activities.

1, 2, 3, 9

Table 28. Public Information and Warning

BSIR ID Project Short Name Project Description Critical

Task 4082, 4083

Establish/enhance citizen awareness of emergency preparedness, prevention, and response measures using SMS messaging

This project requests funds to continue fielding of the ALERT FM system inside the City of Houston. These funds will provide access to SMS (Short Message Service – Text Message) and email connectors, as well as additional ALERT FM devices for schools and public facilities. This will work toward creating a wholly integrated warning solution for the core city in the urban area.

1, 2, 3, 4, 5, 6,

7

98


Task 3957 Establish/enhance

citizen awareness of emergency preparedness by fielding outdoor warning sirens

Logan County will install a total of six Outdoor Warning Sirens. These sirens are needed as follows: two to provide coverage for Zanesfield and Valley Hi (the last two population centers that do not currently have warning sirens), one to replace a failing siren in the Village of Ridgeway, and one each for the Villages of Huntsville, DeGraff, and Quincy to extend coverage to areas of their communities that have a lot of populations that are not adequately covered by their existing sirens. The Logan County Emergency Operations Plan (EOP) addresses all hazards that can affect the county. It has a Basic Plan with Hazard Specific Annexes as well and functional ones. For example, Annex P Ð Terrorism, to the Logan County EOP provides a description of the situation and assumptions, concept of operations, and an organization and assignment of responsibilities for responding to a terrorist incident. The Emergency Management Director is assigned to develop common communications procedures and conduct awareness programs for the public. In response to this directive, Annex C Ð Notification and Warning was developed and addresses the all-hazard use of the outdoor warning system (sirens).

2, 3, 4, 5, 6,

4000 Establish/enhance citizen awareness of emergency preparedness by fielding a long-range acoustic device

Purchase one Long-Range Acoustic Device (LRAD) 100X Battery-Powered Portable Hailing Systems with magnetic mount that can be used for crowd control, evacuations, mass notification, early warning systems, critical infrastructure protection, and control.

2, 3, 4, 5, 6, 7

3949 Establish/enhance a terrorism intelligence/early warning system

ET Continuity of Government (ETCOG) implemented a reverse notification system with Code Red to serve the 14-county region to notify the public of any situation that could arise from a terrorist threat, HazMat threat, or natural disaster where they would need to shelter in place or evacuate or be on alert. This is used in all different aspects of emergency management to warn of danger. This can also be used to alert the public to be on alert for terrorists around them when there is a search. This project is to purchase a reverse notification system and will allow communication to be released to specific areas; it will also allow the public to register their cell phones for notification. Land lines are becoming obsolete and cell phones can be registered for alerts.

1, 2, 3, 4, 5, 6,

7,

99

Table 29. Screening, Search, and Detection


Task 7056 License plate readers

(LPR) Sustainment and expansion of the LPR. The LPR provides automated detection of license plates. The LPR system consists of a high-speed camera, mounted either at a fixed location or on a mobile patrol vehicle, and a computer to convert data from electronic images of vehicle license plates into a readable format, and then compare the information against specified databases of license plates.

1,2,12

7103 Explosive detection canine team

The development, sustainment, and/or enhancement of explosive detection canine team assets. The New York City Police Department will use FY15 funding to enhance explosive detection canine team capabilities through the purchase of equipment and participation in various training opportunities, including detection classes as well as travel to joint regional exercises.

3,4,5,10

7245 Personal protective equipment (PPE) and hand-held explosive detector (EOD)

To purchase PPE and Decontamination Equipment for HazMat Team. This request is for PPE as well as for a hand-held EOD that will be used for response operations by the Irving Police Department EOD Team, which is a Type I EOD.

6,7,8,9

7233 Law enforcement flight operations

This project will sustain and enhance elements of law enforcement flight operations, CI/KR intelligence gathering, and situational screening. The project will include upgrades to cockpit displays for the region’s Metro Air Support Unit, which enhances the Unit’s ability to provide airborne surveillance while protecting critical infrastructure, as well as the purchase of mobile cameras and intelligence gathering software systems for the collection and analysis of telephonic and IP-based communications, which will assist with cyber investigations while sharing information that supports intervention activities.

13

Table 30. Forensics and Attribution


Task 5274 3-D imaging

equipment for crime scene investigation

This forensic package will provide the Burlington County Prosecutor’s Office with the capability of capturing 3-D imaging when investigating various crime scenes. This equipment will enhance investigations concerning homeland security, terrorist, and local criminal activities.

2, 7, 9

100


Task 5265 Enhance radiological

detection capabilities Enhance radiological detection capabilities, including wirelessly mapping radiological readings and utilizing environmental surveillance equipment, including off-site preparedness efforts at surrounding nuclear sites. Nassau County will enhance its HazMat/WMD (CBRNE) Response Team's capabilities through the purchase of CBRNE Response, Detection, and Monitoring Equipment, including but not limited to personnel portal radiation monitors, Radiation Detection Backpacks, handheld FTIR chemical identifiers, and CBRNE PPE.

2, 4, 9

5275, 5276, 5277

Enhance forensic analysis and attribution capabilities

This project seeks to identify and enhance regional forensic analysis and attribute terrorist acts (including the means and methods of terrorism) to their source, including forensic analysis, as well as attribution for an attack and for the preparation for an attack to prevent initial or follow-on acts, and/or swiftly develop counter-options. This will be accomplished by grant funds to support project planning, equipment, and trainings to be determined via a regional subcommittee or workgroup and prioritized according to regional needs. The gaps covered under this core capability include crime scene preservation and exploitation, and evidence collection.

3, 4, 6, 7, 9, 10

B.5 Expert Judgement Elicitation and Script The expert judgement was elicited in two steps. First, the experts were briefed by the HSSEDI Task Leader on the BA model and how their elicitations would be used in the model. Second, each member of the panel was given a set of worksheets containing all the HSGP/Prevention Portfolio project descriptions along with a scoring rubric. The Task Leader read each project description to the entire panel, answered their questions about the description, allowed the members to collaborate with each other, and then asked them to score each project. Two scores were required for each project. The first one was an estimation of the amount the project reduced the probability of impact (loss) given that a terrorist attack had occurred; the second one was an estimation of the amount by which the project reduced the probability of the terrorist attack given that a terrorist threat existed. Each SME individually scored the project by choosing from multiple choice answers that ranged from 1 percent to 100 percent in 5 percent increments. The script used for the expert judgement elicitation can best be described by example. The example chosen is the first project under the first Core Capability, namely the project “Enhance information and vulnerability analysis and information dissemination capabilities (BSIR ID 6114)” under Core Capability “Intelligence and Information Sharing:”

Script: We begin with project “Enhance information and vulnerability analysis and information dissemination capabilities.” Keep in mind that when assessing the project you are not assessing its ability to reduce losses only for the particular

101

locale or jurisdiction in which it was implemented but rather as it, and other similar projects in other locales and states (the project’s “siblings”), contribute to the national Core Capability of Intelligence and Information Sharing in reducing the frequency and impacts of terrorist attacks. I will now read the project description. You may ask me questions to clarify any questions you may have about what the project is. Then you can discuss the project among yourselves for a few minutes. At the end of your group discussion, I ask that you each individually score the project as shown on your worksheets. Namely, score it by what amount the project and its siblings can reduce the probability of impact of terrorist attacks nationwide, and by what amount the project and its siblings reduce the probability of terrorist attacks occurring nationwide. End of script.

102

B.6 Percentage of Dollars Represented by the Sample of Prevention Portfolio Projects The percentage of dollars represented by the Prevention Portfolio sample of projects is:1.5 percent. That percentage was arrived by summing the individual project 2015 budgets and computing the percent of the overall 2015 Prevention Portfolio that sum represents, as shown below.

Table 31. Prevention Portfolio Sample Projects

BSIR ID Project Short Name Grant 3949 Establish/enhance a terrorism intelligence/early warning system $87,623 3957 Establish/enhance citizen awareness of emergency preparedness by fielding

outdoor warning sirens $135,064


$15,000

4082/ 4083

Establish/enhance citizen awareness of emergency preparedness, prevention, and response measures through the use of SMS messaging

$53,730

5265 Enhance radiological detection capabilities $60,000 5274 3-D Imaging equipment for crime scene investigation $51,083 5275/ 5276/ 5277

Enhance forensic analysis and attribution capabilities. $20,000

5931 Continue support of the New York State Intelligence Center (NYSIC) $350,000 6114 Enhance information and vulnerability analysis and information

dissemination capabilities. $545,262

6173 Establish connection between NJ Information Sharing Environment (NJISE) and NJ Data Exchange (NJDEx)

$132,075


$14,000

6404 Enhance explosive ordnance disposal units/bomb squad $127,014 6472 Enhance capability to respond to all-hazards events $185,000 6547 Develop homeland security/emergency management organization and

structure $111,000

6745 Enhance a terrorism intelligence/early warning task force $61,800 7056 License plate readers $1,345,743 7103 Explosive detection canine team $16,000 7233 Law enforcement flight operations $28,480 7245 Personal Protective Equipment and Hand-held Explosive Detector $670

Sample TOTAL $3,339,543

103

Mean 2010-2015 Prevention Portfolio budget: $218,000,000 Percent of Mean: 1.5 percent

The total mean value of all Prevention Portfolio projects for the years 2010 to 2015 is $218,000,000. The total value of the 16 projects selected to represent the Prevention Portfolio is $3,339,543. The selected projects account for an average of 1.5 percent of the total Prevention Portfolio over that six-year period. The 1.5 percent is not the percentage of all the projects similar to the sixteen selected.

104

B.7 SME Responses

Table 32. SME Response: Intelligence and Information Sharing

Projects: SME 1 SME 2 SME 3 SME 4 Average Enhance information and vulnerability analysis and information dissemination capabilities

Attack Low 0.20 0.10 0.20 0.20 0.18 Most Likely 0.80 0.30 0.30 0.30 0.43 High 0.90 0.50 0.50 0.50 0.60

Impact Low 0.50 0.01 0.10 0.10 0.18 Most Likely 0.70 0.20 0.20 0.30 0.35 High 0.80 0.40 0.40 0.40 0.50

Establish connection between NJ Information Sharing Environment (NJISE) and NJ Data Exchange (NJDEx)


Impact Low 0.40 0.01 0.10 0.01 0.13 Most Likely 0.50 0.10 0.30 0.01 0.23 High 0.80 0.10 0.40 0.10 0.35

Establish/enhance a terrorism intelligence/early warning system, center, or task force Attack Low 0.10 0.10 0.01 0.01 0.06

Most Likely 0.30 0.20 0.20 0.10 0.20 High 0.40 0.40 0.30 0.10 0.30

Impact Low 0.10 0.01 0.20 0.01 0.08 Most Likely 0.20 0.10 0.30 0.01 0.15 High 0.20 0.20 0.40 0.10 0.23 Continue support of the New York State Intelligence Center (NYSIC)


105

Projects: SME 1 SME 2 SME 3 SME 4 Average Impact Low 0.10 0.01 0.10 0.01 0.06

Most Likely 0.30 0.10 0.30 0.01 0.18 High 0.50 0.20 0.50 0.10 0.33

Total Core Capability

Average of Projects

Attack Low 0.12 Most Likely 0.30 High 0.43

Impact Low 0.11 Most Likely 0.23 High 0.35

Table 33. SME Response: Interdiction and Disruption

Projects: SME 1 SME 2 SME 3 SME 4 Average Enhance explosive ordnance disposal units/bomb squad

Attack Low 0.10 0.01 0.01 0.01 0.03 Most Likely 0.30 0.01 0.10 0.01 0.11

High 0.50 0.01 0.30 0.10 0.23 Impact Low 0.10 0.01 0.30 0.10 0.13

Most Likely 0.10 0.10 0.40 0.20 0.20

High 0.20 0.10 0.50 0.30 0.28 Enhance capability to respond to all-hazards events


High 0.30 0.01 0.30 0.10 0.18 Impact Low 0.10 0.01 0.20 0.10 0.10

Most Likely 0.20 0.01 0.30 0.20 0.18

High 0.30 0.10 0.40 0.30 0.28

106

Projects: SME 1 SME 2 SME 3 SME 4 Average Develop homeland security/emergency management organization and structure

Attack Low 0.10 0.01 0.20 0.01 0.08 Most

Likely 0.50 0.10 0.40 0.01 0.25

High 0.80 0.30 0.60 0.10 0.45 Impact Low 0.10 0.01 0.10 0.01 0.06

Most Likely 0.40 0.10 0.30 0.10 0.23

High 0.80 0.10 0.50 0.20 0.40 Enhance a terrorism intelligence/early warning task force


High 0.10 0.40 0.30 0.20 0.25 Impact Low 0.01 0.10 0.01 0.01 0.03

Most Likely 0.01 0.20 0.10 0.01 0.08

High 0.10 0.30 0.30 0.10 0.20 Total Core Capability

Average of Projects

Attack Low 0.04 Most Likely 0.15

High 0.28 Impact Low 0.08

Most Likely 0.17

High 0.29

107

Table 34. SME Response: Public Information and Warning

Projects: SME 1 SME 2 SME 3 SME 4 Average Establish/enhance awareness of emergency preparedness, prevention, and response measures

through the use of SMS messaging Attack Low 0.01 0.01 0.01 0.01 0.01

Most Likely 0.10 0.01 0.10 0.01 0.06

High 0.10 0.01 0.20 0.10 0.10 Impact Low 0.01 0.10 0.20 0.01 0.08

Most Likely 0.10 0.20 0.40 0.10 0.20

High 0.30 0.40 0.50 0.10 0.33 Establish/enhance citizen awareness of emergency preparedness by fielding outdoor warning

sirens Attack Low 0.01 0.01 0.01 0.01 0.01

Most Likely 0.01 0.01 0.10 0.01 0.03

High 0.20 0.01 0.30 0.10 0.15 Impact Low 0.01 0.01 0.10 0.01 0.03

Most Likely 0.10 0.10 0.30 0.01 0.13

High 0.10 0.20 0.40 0.10 0.20 Establish/enhance citizen awareness of emergency preparedness by fielding a long-range

acoustic device Attack Low 0.01 0.01 0.10 0.01 0.03

Most Likely 0.01 0.01 0.20 0.01 0.06

High 0.10 0.01 0.40 0.10 0.15 Impact Low 0.01 0.01 0.10 0.01 0.03

Most Likely 0.10 0.10 0.30 0.01 0.13

High 0.30 0.20 0.50 0.10 0.28 Establish/enhance a terrorism intelligence/early warning system


High 0.50 0.01 0.40 0.10 0.25 Impact Low 0.30 0.01 0.20 0.01 0.13

108

Projects: SME 1 SME 2 SME 3 SME 4 Average Most Likely 0.40 0.10 0.40 0.10 0.25


Average of Projects



Most Likely 0.18

High 0.29

Table 35. SME Response: Screening, Search, and Detection

Projects: SME 1 SME 2 SME 3 SME 4 Average License plate readers


High 0.50 0.20 0.40 0.30 0.35 Impact Low 0.10 0.01 0.20 0.01 0.08

Most Likely 0.30 0.20 0.40 0.01 0.23

High 0.50 0.30 0.50 0.10 0.35 Explosive detection canine team


High 0.90 0.30 0.60 0.10 0.48 Impact Low 0.30 0.01 0.20 0.01 0.13

Most Likely 0.50 0.20 0.40 0.10 0.30

High 0.90 0.30 0.50 0.10 0.45 Personal protective equipment and hand-held explosive detector


High 0.50 0.10 0.40 0.10 0.28 Impact Low 0.10 0.01 0.20 0.01 0.08

Most Likely 0.30 0.10 0.40 0.10 0.23

High 0.50 0.20 0.50 0.10 0.33

109

Projects: SME 1 SME 2 SME 3 SME 4 Average Law enforcement flight operations


High 0.60 0.20 0.50 0.10 0.35 Impact Low 0.10 0.01 0.30 0.01 0.11

Most Likely 0.20 0.10 0.50 0.01 0.20


Average of Projects



Most Likely 0.24

High 0.37

Table 36. SME Response: Forensics and Attribution

Projects: SME 1 SME 2 SME 3 SME 4 Average 3-D Imaging equipment for crime scene investigation


High 0.50 0.10 0.40 0.10 0.28 Impact Low 0.10 0.01 0.20 0.01 0.08

Most Likely 0.20 0.10 0.30 0.01 0.15

High 0.60 0.20 0.50 0.10 0.35 Enhance radiological detection capabilities


High 0.40 0.10 0.60 0.10 0.30 Impact Low 0.10 0.01 0.20 0.01 0.08

Most Likely 0.20 0.10 0.50 0.01 0.20

High 0.60 0.10 0.60 0.10 0.35 Enhance forensic analysis and attribution capabilities

Attack Low 0.10 0.01 0.10 0.01 0.06

110

Most Likely 0.10 0.01 0.30 0.10 0.13

High 0.30 0.01 0.40 0.20 0.23 Impact Low 0.10 0.01 0.30 0.01 0.11

Most Likely 0.10 0.01 0.40 0.01 0.13


Average of Projects

Attack Low 0.07 Most Likely 0.14 High 0.27

Impact Low 0.09 Most Likely 0.16 High 0.32

Projects: SME 1 SME 2 SME 3 SME 4 Average

111

B.8 Project Selection Criteria and How They Were FulfilledAs stated in the report, projects were sampled from five core capability areas:

• Intelligence and Information Sharing

• Interdiction and Disruption

• Screening, Search, and Detection

• Forensics and Attribution

• Public Information and WarningThe project sample for each core capability had to meet two selection criteria: 1) each project had to be representative of similar projects under the same core capability that were implemented in at least three other states, and 2) the number of projects in a core capability sample was flexible but was limited by the objective to choose the smallest set that spanned all the key characteristics of each of the core capability. What HSSEDI calls “key characteristics” is meant to be synonymous with what FEMA calls “critical tasks.” The “critical tasks” for each core capability are listed in Table 37.

Table 37. “Critical Tasks” for Each Core Capability

Intelligence and Information Sharing Critical Tasks 1. Planning and Direction: Establish the intelligence and information

requirements of the consumer.a. Rapidly reprioritize law enforcement and intelligence assets as necessary

and appropriate.b. Engage with public and private sector partners in order to determine what

intelligence and information assets may be available for reprioritization.c. Obtain additional information through avenues such as law enforcement

deployment, questioning of witnesses and suspects, increased surveillanceactivity, and community policing and outreach.

2. Collection: Gather the required raw data to produce the desired finishedintelligence and information products.a. Gather/collect information via law enforcement operations, suspicious

activity reporting, surveillance, community engagement, and otheractivities and sources as necessary.

3. Exploitation and Processing: Convert raw data into comprehensibleinformation.

4. Analysis and Production: Integrate, evaluate, analyze, and prepare theprocessed information for inclusion in the finished product.

5. Dissemination: Deliver finished intelligence and information products to theconsumer and others as applicable.a. Develop appropriately classified/unclassified products to disseminate

threat information to local, state, tribal, territorial, federal, international,private sector, nonprofit sector, faith-based organizations, and publicpartners.

6. Feedback and Evaluation: Acquire continual feedback during the intelligencecycle that aids in refining each individual stage and the cycle as a whole.

112

7. Assessment: Continually assess threat information to inform continuedprevention operations and ongoing response activities.

Interdiction and Disruption Critical Tasks 1. Interdict conveyances, cargo, and persons associated with an imminent

terrorist threat or act.2. Prevent terrorist entry into the United States and its territories.3. Prevent movement and operation of terrorists within the United States.4. Render safe and dispose of CBRNE hazards in multiple locations and in all

environments consistent with established protocols.5. Disrupt terrorist financing or prevent other material support from reaching its

target.6. Prevent terrorist acquisition and transfer of CBRNE materials, precursors, and

related technology.7. Conduct antiterrorism operations in the United States.8. Conduct tactical counterterrorism operations in the United States, potentially

in multiple locations and in all environments.9. Strategically deploy assets to interdict, deter, or disrupt threats from reaching

potential target(s).Public Information and Warning Critical Tasks

1. Provide the public with advance notice of a potential terrorist attack againstthe homeland.

2. Update information as an ongoing threat unfolds.3. Must be timely and well-coordinated through standardized procedures4. Inform stakeholders of pending threats.5. Provide instruction on the precautions necessary to protect themselves, their

families, and their property.6. Method of communication with the public should be tailored to best meet the

specific needs of the audience.7. Provide achievable, tangible recovery goals to local and other audiences;

follow up with progress reports as appropriate.Screening, Search, and Detection Critical Tasks

1. Locate persons and networks associated with imminent terrorist threats.2. Develop and engage an observant Nation (i.e., individuals and families;

communities; NGOs; private sector entities; and local, state, tribal, andterritorial partners).

3. Screen and/or scan inbound and outbound persons, baggage, mail, cargo, andconveyances using technical, nontechnical, intrusive, and nonintrusivemeans without unduly hampering commerce.

4. Apply additional measures for high-risk persons, conveyances, or items.5. Conduct physical searches.6. Conduct chemical, biological, radiological, nuclear, and explosive (CBRNE)

surveillance search and detection operations.7. Conduct ambient and active detection of CBRNE.8. Operate in a hazardous environment.9. Conduct technical search/detection operations.10. Conduct nontechnical search/detection operations.

113

11. Conduct bio-surveillance.12. Search databases and other information and intelligence sources.13. Employ wide-area search and detection assets in targeted regions in concert

with state, local, and tribal personnel or other federal agencies (depending onthe threat).

Forensics and Attribution Critical Tasks 1. Preserve the crime scene and conduct site exploitation for intelligence

collection.2. Conduct crime scene investigation.3. Conduct forensic evidence examination, including biometric and DNA

analysis.4. Conduct CBRNE material analysis.5. Conduct digital media, network exploitation, and cyber technical analysis.6. Assess capabilities of perpetrating terrorists and compare with known

terrorist capabilities and methods of operation.7. Conduct investigations to identify the perpetrator(s), conspirator(s), and

sponsorship.8. Interview witnesses, potential associates, and/or perpetrators.9. Analyze intelligence and forensics results to refine/confirm investigative

leads.10. Fuse intelligence, law enforcement information, and technical forensic

conclusions to develop attribution assessments.11. Interpret and communicate attribution results, confidence levels, and their

significance to national decision makers.

114

Using the above Critical Task/Key Characteristics criteria, HSSEDI selected the project samples for each core capability. The resulting project samples, annotated by the Critical Task numbers, are presented in Table 38.

Table 38. Project Samples Annotated with Critical Task Numbers BSIR

ID Project Short

Name Project Description Critical Task

Intelligence and Information Sharing 6114 Enhance

information and vulnerability analysis and information dissemination capabilities

This project enhances information analysis and law enforcement capabilities. This project includes upgrades to computers and software and training to more effectively gather and share information. For example, the project includes ALPR for identifying possible threats, Los Angeles Regional Common Operational Picture Program to electronically gather and transmit data to off-site stakeholders, Video Downlink that sets up digital high-definition video link systems serving multiple agencies, and Palantir equipment upgrade from various agencies to expand capabilities for better intelligence analysis and information sharing. This project supports Law Enforcement Patrol Team and Law Enforcement Aviation-Helicopters-Patrol and Surveillance NIMS Typed Resource as well as other state and local typed resource.

1, 2, 5

6173 Establish connection between NJ Information Sharing Environment (NJISE) and NJ Data Exchange (NJDEx)

This project will allow for the connection of the NJDEx system to the NJISE. The NJISE enables the NJ law enforcement community to search and share police reports with participating police departments statewide. NJDEx provides critical support to the state’s investigative and analytical efforts as well. Additionally, NJDEx shares information with the FBI’s National Data Exchange (NDEx) program. Incorporating the NJDEx system into the NJISE environment provides a solution that allows for data interoperability services, bundled searches and network services. The network services include NJDEX, HSIN, LEO, RISS, and others, as well as application services over time as part of an iterative process.

2, 5


This project will maintain a statewide Suspicious Activity Reporting (SAR) network for a holistic view of terrorism and criminal-related suspicious activity in Texas. This network facilitates comprehensive collection, analysis, and response to terrorism and crime-related activity between identified fusion centers in Texas participating in the network. This network will assist federal, state, tribal, and local law enforcement agencies in their efforts to detect, deter, and eliminate criminal activities. The Texas Joint Crime Information Center (TX JCIC) is utilizing TripWire Threat Detection and Analysis System for the state’s internal SAR program.

1, 2, 3, 4

115

BSIR ID

Project Short Name Project Description Critical

Task 5931 Continue support

of the New York State Intelligence Center (NYSIC)

The New York State Police (NYSP) will utilize funds to support the NYSIC, which is the Governor’s designated Fusion Center in the state. The NYSIC will leverage grant funding to support the Phase 2 development of its Criminal Intelligence Analysis System (CIAS), for Intelligence Analysts, to support the Field Intelligence Officer (FIO) Program (to engage local law enforcement), and for supporting planning, training, and travel costs. These efforts will support all four COCs (Receive, Analyze, Disseminate, Gather) and the EC of Communications and Outreach. Also, the projects will sustain strengths in the NYSIC’s Fusion Center Assessment, as well as addressing gaps related to analyst training.

6, 7

Interdiction and Disruption 6404 Enhance explosive

ordnance disposal units/bomb squad

Dekalb County Police Department will use FY2015 HSGP funding in the amount of $127,014 to sustain and maintain current Explosive Ordnance Disposal (EOD) program. Funding will be used to purchase needed equipment and EOD/IED training devices for terrorism prevention and to sustain existing bomb disposal unit (BDU) teams through the repair and replacement of worn equipment, upgrades to current equipment, and the purchasing of new equipment as technology advances.

4

6472 Enhance capability to respond to all-hazards events

Glendale Police Department Special Operations Division responds to all CBRNE and Tactical calls for service within the city limits as well as neighboring jurisdictions when called upon. The division currently has no specialized vehicle in which to work command elements in the event of a CBRNE / Tactical situation involving extended crisis negotiations or communications support. The division has the necessary equipment, personnel, and training for these events, but lacks a specialized vehicle in which to contain and work. This project would entail the building and outfitting of a specialized vehicle to be used in these situations to enhance the capability that already exists. As a city that host very large, high-profile events, the necessity of such a vehicle that is maneuverable, and mobile has become very apparent.

1, 4, 5, 6, 7, 8,

9

6547 Develop homeland security/ emergency management organization and structure

Region 6 plans to equip, train, and exercise response personnel and specialized teams to effectively respond to violent extremism and other hazards/incidents/Search and Rescue Operations. Funds will be utilized to sustain, maintain, and enhance local and regional CBRNE and All Hazard Response Capabilities, specialized teams, specialized support teams, operational interoperability, and operational coordination of capabilities and resources. This project will also provide funding to address gaps in Mass Search and Rescue Operations; On-Scene Security and Protection activities will address gaps in planning, equipment, training, and exercises to protect response personnel; securing disaster areas; and law enforcement. Multiple agencies, jurisdictions, and disciplines may submit projects under this investment.

1, 5, 7, 8, 9

116

BSIR ID


Task 6745 Enhance a

terrorism intelligence/early warning task force

The Texas Rangers Special Operations Group (TRSOG) requires the capability to integrate state law enforcement ATV activities with federal partners in order to respond to, track, and trail criminals or lost subjects along the Texas-Mexico border. This capability will allow TRSOG to provide accurate distances from observation posts to friendly ground, mobile, and air elements to enable the disruption, deterrence, and denial of transnational criminal smuggling activities.

1, 2, 3, 9

Public Information and Warning 4082, 4083

Establish/enhance citizen awareness of emergency preparedness, prevention, and response measures through the use of SMS messaging

This project requests funds to continue fielding of the ALERT FM system inside the City of Houston. These funds will provide access to SMS (Short Message Service – Text Message) and email connectors, as well as additional ALERT FM devices for schools and public facilities. This will work toward creating a wholly integrated warning solution for the core city in the urban area.

1, 2, 3, 4, 5, 6,

7

3957 Establish/enhance citizen awareness of emergency preparedness by fielding outdoor warning sirens

Logan County will install a total of six Outdoor Warning Sirens. These sirens are needed as follows: two to provide coverage for Zanesfield and Valley Hi (the last two population centers that do not currently have warning sirens), one to replace a failing siren in the Village of Ridgeway, and one each for the Villages of Huntsville, DeGraff, and Quincy to extend coverage to areas of their communities that have a lot of outdoor populations that are not adequately covered by their existing sirens. The Logan County Emergency Operations Plan (EOP) addresses all hazards that can affect the county. It has a Basic Plan with Hazard Specific Annexes, as well and functional ones. Annex P Ð Terrorism, to the Logan County EOP provides a description of the situation and assumptions, concept of operations, and an organization and assignment of responsibilities for responding to a terrorist incident. The Emergency Management Director is assigned to develop common communications procedures and conduct awareness programs for the public. In response to this directive, Annex C Ð Notification and Warning, was developed and addresses the all-hazard use of our outdoor warning system (sirens).

2, 3, 4, 5, 6,


Purchase one (1) Long-Range Acoustic Device (LRAD) 100X Battery-Powered Portable Hailing Systems with magnetic mount that can be used for crowd control, evacuations, mass notification, early warning systems, critical infrastructure protection, and control.

2, 3, 4, 5, 6, 7

117

BSIR ID


Task 3949 Establish/enhance

a terrorism intelligence/early warning system

ETCOG implemented a reverse notification system with Code Red to serve the 14-county region to notify the public of any situation that could arise from a terrorist threat, HazMat threat, or natural disaster where they would need to shelter in place or evacuate or be on alert. This is used in all different aspects of emergency management to warn of danger. This can also be used to alert the public to be on alert for terrorists around them when there is a search. This project is to purchase a reverse notification system and will allow communication to be released to specific areas and will also allow the public to register their cell phones for notification. Land lines are becoming obsolete and cell phones can be registered for alerts.

1, 2, 3, 4, 5, 6,

7,

Screening, Search, and Detection 7056 License plate

readers Sustainment and expansion of the LPR. The LPR provides automated detection of license plates. The LPR system consists of a high-speed camera, mounted either at a fixed location or on a mobile patrol vehicle, and a computer to convert data from electronic images of vehicle license plates into a readable format, and then compare the information against specified databases of license plates.

1,2,12

7103 Explosive detection canine team

The development, sustainment and/or enhancement of explosive detection canine team assets. The New York City Police Department will use FY15 funding to enhance explosive detection canine team capabilities through the purchase of equipment and participate in various training opportunities, including detection classes as well as travel to joint regional exercises.

3,4,5, 10

7245 Personal protective equipment and hand-held explosive detector

To purchase Personal Protective Equipment and Decontamination Equipment for HazMat Team. This request is for personnel protective equipment (PPE) as well as a hand-held explosive detector that will be used for response operations by the Irving Police Department EOD Team, which is a Type I EOD.

6,7,8,9

7233 Law enforcement flight operations

This project will sustain and enhance elements of law enforcement flight operations, CI/KR intelligence gathering, and situational screening. The project will include upgrades to cockpit displays for the region’s Metro Air Support Unit, which enhances the Unit’s ability to provide airborne surveillance while protecting critical infrastructure; the purchase of mobile cameras and intelligence gathering software systems for the collection and analysis of telephonic and IP-based communications, which will assist with cyber investigations while sharing information that supports intervention activities.

13

118

BSIR ID


Task Forensics and Attribution

5274 3-D Imagingequipment forcrime sceneinvestigation

This forensic package will provide the Burlington County Prosecutor's Office with the capability of capturing 3-D imaging when investigating various crime scenes. This equipment will enhance investigations concerning homeland security, terrorist, and local criminal activities.

2, 7, 9

5265 Enhance radiological detection capabilities

Enhance radiological detection capabilities, including wirelessly mapping radiological readings and utilizing environmental surveillance equipment, including off-site preparedness efforts at surrounding nuclear sites. Nassau County will enhance its HazMat/WMD (CBRNE) Response Team's capabilities through the purchase of CBRNE Response, Detection, and Monitoring Equipment, to include but not limited to personnel portal radiation monitors, Radiation Detection Backpacks, handheld FTIR chemical Identifiers and CBRNE PPE.

2, 4, 9

5275, 5276, 5277

Enhance forensic analysis and attribution capabilities

This project seeks to identify and enhance regional forensic analysis and attribute terrorist acts (including the means and methods of terrorism) to their source, including forensic analysis, as well as attribution for an attack and for the preparation for an attack in an effort to prevent initial or follow-on acts and/or swiftly develop counter-options. This will be accomplished by grant funds to support project planning, equipment, and trainings to be determined via a regional subcommittee or workgroup and prioritized according to regional needs. The gaps covered under this core capability include crime scene preservation and exploitation, and evidence collection.

3, 4, 6, 7, 9, 10

119

B.9 Net Benefit Cost Ratio for Core Capability

Figure 18. Mapping of Intelligence and Information Sharing Net Benefit Cost Ratio Range

Figure 19. Mapping of Interdiction and Disruption Net Benefit Cost Ratio Range

120

Figure 20. Mapping of Screening, Search, and Detection Net Benefit Cost Ratio Range

Figure 21. Mapping of Public Information and Warning Net Benefit Cost Ratio Range

121

Figure 22. Mapping of Forensics and Attribution Net Benefit Cost Ratio Range

The Forensics and Attribution core capability is nearly all green, which represents a NBCR of 5 or greater (see colored labels on top of the graph). This colorization has to do with a single attribute of the core capability: its mean budget value (for 2010-2015). It is not reflective of any other core capability attribute. The smaller a core capability’s mean budget, the less far along the x- and y-axes one must move to reach the green region on the chart. Since the Forensics andAttribution core capability has the lowest budget of any core capability, even small reductions inimpact or reduction in attack frequency put it in the green region.

122

List of Acronyms

AI Artificial Intelligence

BA Breakeven Analysis

BCA Benefit Cost Analysis

BSIR Biannual Strategy Implementation Report

BTR Baseline Annual U.S. Exposure to Terrorism

CBA Cost-Benefit Analysis

CBP Customs and Border Protection

CBRNE Chemical, Biological, Radiological, Nuclear, and Explosives

CDI Conditional Distribution of Injuries

CDF Conditional Distribution of Fatalities

CDP Conditional Distribution of Property Damage

CIAS Criminal Intelligence Analysis System

COA Course of Action

DHS Department of Homeland Security

DOM Detailed Operations Model

DOT Department of Transportation

EMS Emergency Management Service

EOD Explosive Ordinance Disposal

EOP Emergency Operations Plan

FEMA Federal Emergency Management Agency

FFI Fraction of Events that Cause Injury or Fatality

FFRDC Federally Funded Research and Development Center

FIO Field Intelligence Officer

FIPS Federal Information Processing Standard

123

FL Florida

FPD Fraction of Events that Cause Property Damage

FT Frequency of Terror Attacks Per Year

FY Fiscal Year

GTD Global Terrorism Database

HAZMAT Hazardous Material

HAZUS-MH Hazards in the United States – Multi-Hazard

HIFLD Homeland Infrastructure Foundation-Level Data

HSGP Homeland Security Grants Program

HSSEDI Homeland Security Systems Engineering and Development Institute

IJ Investment Justification

LPR License Plate Reader

LRAD Long Range Acoustic Device

MAUT Multi-Attribute Utility Theory

MFI Conditional Monetized Value of Fatalities and Injuries

NDEx National Data Exchange

NBCR Net Benefit Cost Ratio

NJ New Jersey

NJISE New Jersey Information Sharing Environment

NOAA National Oceanic and Atmospheric Administration

NPAD National Preparedness Assessment Division

NYSIC New York State Intelligence Center

NYSP New York State Police

OMB Office of Management and Budget

OPSG Operation Stonegarden

POETE Planning, Organization, Equipment, Training, and Exercises

124

PPE Personnel Protective Equipment

QOL Quality of Life

ROI Return on Investment

RPA Revealed Preference Analysis

SAR Search and Rescue

SDM Senior Decision Makers

SHSP State Homeland Security Program

SLTT State, Local, Tribal, and Territorial

SME Subject Matter Expert

SMS Special Message Service

SPR State Preparedness Report

THIRA Threat and Hazard Identification and Risk Assessment

TRSOG Texas Rangers Special Operations Group

TX JCIC Texas Joint Crime Information Center

UASI Urban Area Security Initiative

USCG United States Coast Guard

VDEM Virginia Department of Emergency Management

VI Fractional Value of Statistical Life of Injury

VMASC Virginia Modeling and Simulation Center

VSL Value of Statistical Life

WTP Willingness to Pay

125

List of References U.S. Department of Homeland Security (DHS). Fiscal Year 2017 Homeland Security Grant Program (HSGP), Notice of Funding Opportunity (NOFO). [https://www.fema.gov/media-library/assets/documents/131992]

U.S. Department of Homeland Security, National Preparedness Goal, Second Edition. September 2015. [https://www.fema.gov/national-preparedness-goal]

U.S. Department of Homeland Security, National Preparedness Directorate, 10 October 2017. [https://www.fema.gov/national-preparedness-directorate]

HSSEDI, 23 March 2018, Phase 1 FEMA NPAD Homeland Security Grant Program Return on Investment Methodology.

Office of Management and Budget, Circular A-94: Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs, Transmittal Memo No. 64.

Ewing, P.L., Jr., W. Tarantino, and G.S. Parnell, “Use of Decision Analysis in the Army Base Realignment and Closure (BRAC) 2005 Military Value Analysis,” Decision Analysis, 3.1. 33(17); DOI:10.1287/deca.1060.0071, March 2006.

Feng, T. and Keller, L.R., “A Multiple-Objective Decision Analysis for Terrorism Protection: Potassium Iodide Distribution in Nuclear Incidents,” Decision Analysis, 3.2 (June 2006): 76(18); doi:10.1287/deca.1060.0072.

Golab, Kamal; Kirkwood, Craig W.; Sicherman, Alan. February 1981. Selecting a Portfolio of Solar Energy Projects Using Multi-Attribute Preference Theory. Management Science, Vol. 27, No. 2, pp. 174-189.

Shreve, C.M., and I. Kelman, 2014, “Does mitigation save? Reviewing cost-benefit analyses of disaster risk reduction,” International Journal of Disaster Risk Reduction, 10, pp. 213–235.

Virginia Department of Emergency Management, “Governor McAuliffe Announces $5.7 Million in Homeland Security Grant Awards,” 3 October 2016. [http://www.vaemergency.gov/governor-mcauliffe-announces-5-7-million-homeland-security-grant-awards/]

Ezell, B., Lawsure, K., and Flanagan, D., “Risk and Decision Analytic Support to the Commonwealth of Virginia State Homeland Security Program, Final Report,” Virginia Modeling, Analysis and Simulation Center, 31 December 2015, updated 2 February 2016.

Rose, A. et al., “Benefit-Cost Analysis of FEMA Hazard Mitigation Grants,” DOI: 10.1061/(ASCE)1527-6988-(2007)8:4(97).

U.S. Census Bureau, American Community Survey (ACS), 2011-2015. [https://www.census.gov/geo/maps-data/data/tiger-data.html]

U.S. Department of Homeland Security, Homeland Infrastructure Foundation-Level Data (HIFLD). [https://hifld-geoplatform.opendata.arcgis.com/]

https://www.fema.gov/media-library/assets/documents/131992

https://www.fema.gov/media-library/assets/documents/131992

https://www.fema.gov/national-preparedness-goal

https://www.fema.gov/national-preparedness-directorate





fema national preparedness assessment division (npad ......convenient data set that enabled hssedi...

Documents