gao-19-172, small community air service ...since fiscal year 2002, dot has awarded 401 scasdp grants...

63
SMALL COMMUNITY AIR SERVICE DEVELOPMENT Process for Awarding Grants Could Be Improved Report to Congressional Requesters March 2019 GAO-19-172 United States Government Accountability Office

Upload: others

Post on 04-Jul-2020

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

SMALL COMMUNITY AIR SERVICE DEVELOPMENT

Process for Awarding Grants Could Be Improved

Report to Congressional Requesters

March 2019

GAO-19-172

United States Government Accountability Office

Page 2: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

United States Government Accountability Office

Highlights of GAO-19-172, a report to congressional requesters

March 2019

SMALL COMMUNITY AIR SERVICE DEVELOPMENT Process for Awarding Grants Could Be Improved

What GAO Found Some aspects of the Department of Transportation’s (DOT) process for evaluating fiscal year 2014–2016 grant applications for the Small Community Air Service Development Program (SCASDP) were inconsistent with its published grant notices, which communicate the process for potential applicants, and with its internal evaluation plan, which is used by reviewers to rate applications. In addition, DOT followed or partially followed recommended practices for awarding discretionary grants.

• Grant notice and evaluation plan: DOT’s process for evaluating application eligibility and merit differed from the process described in its grant notices. For example, DOT’s notice stated that it would use the criteria that airports have either insufficient air service or unreasonably high airfares to determine whether an application is eligible for a grant, but in practice, DOT used these criteria to evaluate an application’s merit. According to Office of Management and Budget guidance, the grant notice should make the application process transparent. In response to GAO’s finding, DOT is revising its upcoming grant notice to clarify how it uses information submitted by applicants, which provides greater transparency of its process. Further, DOT’s internal evaluation plan for reviewers described the selection criteria to consider, but did not provide clear guidance for evaluating and rating each application based on how it aligns with these criteria. DOT’s financial award guidance calls for a clearly defined application evaluation process to enable reviewers to rate applications fairly and accurately. Clarifying its evaluation guidance could help ensure that reviewers consistently rate applications. DOT officials told GAO that they intend to revise the evaluation plan for the next grant cycle to address these issues, but as of March 2019, have not provided documentation that such changes are in progress.

• Recommended practices: DOT followed the practices of using a panel of reviewers with expertise to evaluate applications and also documenting its rationale for each award decision, including providing a written narrative describing the grant project and outlining how the project aligned with the selection criteria. However, while DOT notified applicants of award decisions through its grant award order, unsuccessful applicants GAO interviewed were not always aware that they could request feedback from DOT. In response to GAO’s finding, DOT is adding information on the opportunity for feedback to its upcoming grant notice. This change should help make applicants aware of this opportunity and could help unsuccessful applicants determine whether to apply again, and if so, how to improve their applications.

Overall, GAO found that half of the 66 fiscal year 2010–2014 grant projects reviewed were successful in reaching their project goals, and over one-third sustained their air service improvements at least 24 months after their grant ended. GAO found that factors including community demand for air service, a strong or growing local economy, and airline support for a project were associated with project success; however, no single factor strongly correlated with project success.

View GAO-19-172. For more information, contact Heather Krause, 202-512-2834, [email protected].

Why GAO Did This Study Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO was asked to review DOT’s award process and the effectiveness of recent grants.

This report, among other things, (1) examines the extent to which DOT’s process for awarding fiscal year 2014–2016 grants (the most recent award cycles when GAO began its review) was consistent with its grant notices and recommended practices for awarding discretionary grants, and (2) examines the extent to which fiscal year 2010–2014 grants (the most recent award cycles for which most projects had been completed) assisted airports in improving their air service, and identifies factors that affect the success of grant projects. GAO reviewed program documentation; compared processes against internal documents and recommended practices that GAO identified in previous work; analyzed calendar year 2009–2019 airline and DOT data, conducted a correlation analysis; and interviewed DOT officials, and a judgmental sample of 36 grantee or applicant airports and 13 stakeholders in small community air service.

What GAO Recommends GAO recommends that DOT clarify in its internal evaluation plan how reviewers should evaluate and rate applications. DOT agreed with the recommendation.

Page 3: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page i GAO-19-172 Small Community Air Service Development

Letter 1

Background 5 DOT’s Award Process Was Inconsistent with Its Grant Notices

and Lacked Clear Evaluation Guidance, but Was Consistent with Some Recommended Practices for Awarding Grants 14

DOT Collects Information to Oversee Grant Projects and Monitors Grantee Performance in Establishing or Expanding Air Service 25

Half of Fiscal Year 2010–2014 Grant Projects Were Successful in Achieving Their Goals, Which Selected Airports and Stakeholders Attributed to Several Factors 27

Conclusions 37 Recommendation for Executive Action 38 Agency Comments 38

Appendix I Objectives, Scope, and Methodology 40

Appendix II Federal Agencies, Airports, and Stakeholders in Small Community Air Service GAO Interviewed 53

Appendix III Comments from the Department of Transportation 55

Appendix IV GAO Contact and Staff Acknowledgments 57

Tables

Table 1: Commercial Air Service Changes in the United States by Airport Size, 2007–2016 6

Table 2: Requirements for Applicants and Projects for Small Community Air Service Development Program ( SCASDP) Grants, Fiscal Years 2014–2016 10

Table 3: Selection Criteria for Small Community Air Service Development Program (SCASDP) Grants, Fiscal Years 2014–2016 11

Contents

Page 4: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page ii GAO-19-172 Small Community Air Service Development

Table 4: Rating Definitions for Small Community Air Service Development Program (SCASDP) Grant Applications 20

Table 5: GAO’s Assessment of the Extent to Which the Small Community Air Service Development Program (SCASDP) Followed Recommended Practices for Awarding Discretionary Grants (Fiscal Years 2014–2016) 22

Table 6: Evaluation Results for Closed Small Community Air Service Development Program (SCASDP) Grants, Fiscal Years 2010-2014 29

Table 7: Factors Affecting the Success of Small Community Air Service Development Projects and Measures 32

Table 8: GAO’s Categories and Definitions for Small Community Air Service Development Program Grants 43

Table 9: GAO’s Metrics of Success for “Establish Service” Grants 44 Table 10: GAO’s Metrics of Success for “Promote Service”

Grants with Comparison Airports 46 Table 11: GAO’s Metrics of Success for “Promote Service” Grants

without Comparison Airports 47 Table 12: GAO’s Metrics of Success for “Expand Service” Grants 48 Table 13: Federal Agencies, Airports, and Stakeholders in Small

Community Air Service GAO Interviewed 53

Figures

Figure 1: Available Funding and Number of Grants for the Small Community Air Service Development Program, Fiscal Years 2002–2017 9

Figure 2: The Department of Transportation’s Process for Evaluating Small Community Air Service Development Program Grant Applications, Fiscal Years 2014–2016 12

Figure 3: Airports That Received Small Community Air Service Development Program Grants, Fiscal Years 2010–2014 28

Figure 4: Comparison of Connectivity Change for a Fiscal Year 2011 Small Community Air Service Development Program Grantee 35

Page 5: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page iii GAO-19-172 Small Community Air Service Development

Abbreviations DOT Department of Transportation EAS Essential Air Service FAA Federal Aviation Administration IATA International Air Transport Association MIT Massachusetts Institute of Technology OIG Office of Inspector General OMB Office of Management and Budget SCASDP Small Community Air Service Development Program

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.

Page 6: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 1 GAO-19-172 Small Community Air Service Development

441 G St. N.W. Washington, DC 20548

March 26, 2019

The Honorable Peter A. DeFazio Chairman The Honorable Sam Graves Ranking Member Committee on Transportation and Infrastructure House of Representatives

The Honorable Rick Larsen Chairman Subcommittee on Aviation Committee on Transportation and Infrastructure House of Representatives

For at least the last two decades, small airports and the communities they serve have faced challenges in attracting and retaining commercial air service. According to a Massachusetts Institute of Technology (MIT) study, small community airports have been particularly affected by reductions in scheduled domestic flights from major airlines, and a decrease in the use of small regional jets that serve small community airports.1 Further, our previous work found that growing service from low-cost airlines has contributed to passengers driving to relatively close large airports to obtain lower airfares rather than use their local airport, thus reducing demand for service at small airports.2 From 2007 through 2016, available domestic flights at smaller U.S. airports fell by 31.5 percent, as compared to a 6.2 percent decline in flights at the largest airports.3

Congress established the Small Community Air Service Development Program (SCASDP) in 2000 to help underserved airports improve their air

1Michael Wittman and William Swelbar, Trends and Market Forces Shaping Small Community Air Service in the United States, (MIT International Center for Air Transportation, May 2013). Mainline airlines provide domestic and international passenger and cargo service on larger aircraft. Regional airlines provide domestic and limited international passenger service, generally using aircraft with fewer than 90 seats, and cargo service to smaller airports. 2GAO, Commercial Aviation: Status of Air Service to Small Communities and the Federal Programs Involved, GAO-14-454T (Washington, D.C.: Apr. 30, 2014). 3Department of Transportation, Report of the Working Group on Improving Air Service to Small Communities (May 2017).

Letter

Page 7: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 2 GAO-19-172 Small Community Air Service Development

service through discretionary grants.4 The program is administered by the Office of Aviation Analysis in the Department of Transportation’s (DOT) Office of the Secretary. The process of evaluating grant applications is key in selecting which of the proposed projects will receive awards. DOT makes applicants aware of this process in its grant notices and provides an internal plan to its staff to guide these evaluations. DOT has awarded approximately $188 million to more than 400 communities since the program’s inception; however, questions remain about the extent to which grants have been successful in helping small communities to improve air service. A 2008 study found that of the 40 fiscal year 2002–2006 grant projects reviewed, 30 percent were successful in achieving their goals and sustained the resulting benefits for at least 12 months.5 Another study from 2014 found that approximately 37 percent of the 115 fiscal year 2006–2011 grants reviewed were successful in providing new scheduled service or marketing existing service for at least 28 months after the grant was awarded.6

You asked us to review DOT’s award process and assess the effectiveness of recent grant projects to ensure that SCASDP is fulfilling its goal to improve air service to underserved airports. This report: (1) examines the extent to which DOT’s process for awarding fiscal year 2014–2016 SCASDP grants was consistent with its grant notices, internal evaluation plan, and recommended practices for discretionary grant programs; (2) identifies steps DOT has taken to oversee grant projects and monitor their performance; and (3) examines the extent to which fiscal year 2010–2014 grants assisted airports in improving their air service, and identifies factors that affect the success of grant projects.

To examine the extent to which DOT’s grant award process is consistent with its grant notices and evaluation plan, we first identified DOT’s process for awarding fiscal year 2014–2016 grants through a review of documentation including the three public notices for these grants, DOT’s evaluation of grant applications, award orders announcing fiscal year

4Wendell H. Ford Aviation Investment and Reform Act for the 21st Century, Pub. L. 106–181, title II, § 203(a), 114 Stat. 92 (2000). 5Office of Inspector General (OIG), DOT, The Small Community Air Service Development Program, CR-2008-051 (May 13, 2008). 6Michael D. Wittman, Public Funding of Airport Incentives: The Efficacy of the Small Community Air Service Development Grant Program, MIT International Center for Air Transportation (January 2014).

Page 8: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 3 GAO-19-172 Small Community Air Service Development

2014–2016 grantees, and statements made by DOT officials regarding implementation of the grant program.7 We evaluated DOT’s award process for fiscal year 2014–2016 grants because they were the three most recent grant award cycles completed when we began our audit work. We then compared DOT’s actual award process to the process for evaluating grant applications for eligibility and technical merit described in DOT’s grant notices for fiscal years 2014–2016 and to the evaluation plan DOT used to guide the evaluation of grant applications. We also compared DOT’s award process to guidance for administering grants from the Office of Management and Budget (OMB) and DOT.8 We interviewed a non-generalizable sample of six successful and six unsuccessful applicants for fiscal year 2014–2016 grants—selected to reflect a mix of grant years, airport hub sizes, and types of projects, among other factors—to obtain their perspectives on DOT’s award process. The views of these applicants cannot be generalized to all SCASDP applicants. We also interviewed representatives from four aviation consulting firms to obtain their perspectives on DOT’s award process and learn how they assist communities in submitting applications. We selected these firms because they provided assistance to grantees within our scope of work. To determine the extent to which DOT followed recommended practices for awarding discretionary grants, we compared the attributes of five practices we previously identified to DOT’s fiscal year 2014–2016 award process, as described above.9

To identify the steps DOT has taken to oversee grant projects and monitor their performance, we reviewed selected quarterly project status reports and all final reports submitted to DOT by fiscal year 2010–2014

7DOT announced the communities receiving fiscal year 2017 grants in July 2018, and we did not include those grants in the scope of our review. As of March 2019, DOT has not issued a public notice for the fiscal year 2018 grant award cycle. 82 C.F.R. Part 200. DOT, Financial Assistance Guidance Manual (December 2016). 9GAO, Intercity Passenger Rail: Recording Clearer Reasons for Award Decisions Would Improve Otherwise Good Grantmaking Practices, GAO-11-283 (Washington, D.C.: Mar. 10, 2011). We previously identified a total of six recommended practices, including to (1) communicate with potential applicants prior to the competition; (2) develop a technical review panel with certain characteristics; (3) assess applicants’ capabilities to account for funds; (4) plan for administering the technical review; (5) notify applicants of awards decisions; and (6) document the rationale for awards decisions. We did not apply the practice related to assessing applicants’ ability to account for funds in our evaluation because SCASDP grants are disbursed on a reimbursable basis, and therefore, communities make the initial expenditures for projects prior to seeking reimbursement from DOT.

Page 9: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 4 GAO-19-172 Small Community Air Service Development

grantees, and interviewed DOT program officials on their processes for monitoring and evaluating grant projects. Additionally, we interviewed airport officials from 20 communities that received grants during this time period about their communication with DOT regarding project progress. We selected these communities using a stratified, random sample.

To examine the extent to which the fiscal year 2010–2014 grants assisted airports in improving their air service, we evaluated the extent to which grantees were successful in accomplishing their individual project goals stated in their grant applications or grant agreements. We evaluated fiscal year 2010–2014 grant projects because they represented the most recent grant award cycles—i.e., the time period from issuance of the grant notice to the announcement of selected applicants—for which the majority of projects had been completed when we began our audit work. We did so by analyzing airline schedule and DOT flight data from calendar year 2009 through 2019 including the number of scheduled departures and passenger enplanements, and the type of aircraft used. We obtained these data from Diio, a private contractor that provides online access to U.S. airline financial, operational, and passenger data. We determined that the data were sufficiently reliable for our purposes by reviewing the quality control procedures used by Diio and interviewing DOT officials responsible for data collection efforts. We then determined whether individual projects were unsuccessful, partially successful, or successful in accomplishing their goals to establish, promote, or expand their air service. For projects we deemed successful, we also evaluated the extent to which their success was sustained after the grant ended. See appendix I for more details on our methodology, including how we determined project success.

For our last objective, we also evaluated whether fiscal year 2010–2014 grantees experienced an increase in connectivity, a measure of an airport’s degree of access to the global air transportation system, during the grant period. We compared the change in connectivity at each grantee airport—the difference between its pre-grant connectivity and its connectivity at the end of the grant period—to the corresponding changes at a comparison group of similarly-sized airports. If a grantee airport’s change in connectivity was at least 5 percentage points greater than the equivalent change in the average measures for its comparison group, we determined that it experienced an increase in connectivity.

Finally, to identify factors affecting grant project success, we interviewed officials from the 20 selected fiscal year 2010–2014 grantee airports identified above, as well as officials from four grantee airports where we

Page 10: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 5 GAO-19-172 Small Community Air Service Development

also conducted site visits, and a judgmental sample of 13 stakeholders in small community air service—including three academic experts and researchers, and representatives from two industry associations, four aviation consulting firms, and four airlines. We identified and selected these stakeholders based on their expertise in their fields. Their views cannot be used to make generalizations about the views of all stakeholders in small community air service, but do provide a range of perspectives on issues affecting the industry. See appendix II for a list of interviewees. We also performed a correlation analysis for grantees whose project goals were to establish or promote service to determine whether any of the factors grantees identified correlated with successful grants.

We conducted this performance audit from July 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Communities of all sizes seek access to air service as a driver for attracting investment, generating employment, and providing mobility for citizens. According to the Federal Aviation Administration (FAA), aviation is important to economic performance because it supports economic output, attracts business and tourism, supports local economic development, and helps retain jobs that might otherwise be relocated elsewhere.10 Small communities in particular can obtain economic benefits from connection to the global air transportation network.11 For

10Federal Aviation Administration, The Economic Impact of Civil Aviation on the U.S. Economy: Economic Impact of Civil Aviation by State (Washington, D.C.: September 2017). 11No common definition exists for what constitutes a “small community”; however, for SCASDP eligibility pursuant to criteria in federal statute, a community must be served by an airport no larger than a small hub airport. 49 U.S.C § 41743(c)(1).

Background

Page 11: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 6 GAO-19-172 Small Community Air Service Development

instance, direct service to a mainline airline’s hub can provide one-stop access to hundreds of additional destinations around the globe.12

However, we reported in 2014 that several factors have contributed to a decline in air service to small communities, including higher jet fuel costs, declining population levels in those communities, and a potential shortage of qualified pilots.13 As shown in table 1, since 2007, small and non-hub airports have experienced a decrease in air service, as measured by scheduled departures and available seats, particularly in comparison with large hub airports.

Table 1: Commercial Air Service Changes in the United States by Airport Size, 2007–2016

Hub size % Change in departures % Change in seats Large -6.2% +6.6% Small and Non-Hub -31.5% -16.9%

Source: Department of Transportation, Report of the Working Group on Improving Air Service to Small Communities. | GAO-19-172

Additionally, recent research has attributed a number of challenges that small communities face in maintaining commercial air service to industry dynamics resulting from airline consolidation. As we reported in 2014, the financial crisis and recession from 2007 through 2009 led to heavy financial losses for mainline airlines and contributed to a wave of bankruptcies and airline mergers.14 In 2017, the four largest airlines—American Airlines, Delta Air Lines, Southwest Airlines, and United Airlines—carried 80 percent of domestic travelers.15 In the wake of the

12Airport hub categories are based on the number of passengers boarding an aircraft (enplanements) for all operations of U.S. airlines in the United States. A nonhub airport enplanes less than 0.05 percent of all passengers, a small hub enplanes 0.05–0.249 percent of all passengers, a medium hub enplanes 0.25–1 percent of all passengers, and a large hub airport enplanes more than 1 percent of all passengers. 49 U.S.C. § 40102 (a)(29), (31), (34), and (42). 13GAO-14-454T. 14GAO, Airline Competition: The Average Number of Competitors in Markets Serving the Majority of Passengers Has Changed Little in Recent Years, but Stakeholders Voice Concerns about Competition, GAO-14-515 (Washington, D.C.: June 11, 2014). 15Russell Mills and Nicole Kalaf-Hughes, “The Importance of Markets, Politics, and Community Support: An Analysis of the Small Community Air Service Development Program,” Journal of Air Transport Management (September 2017).

Page 12: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 7 GAO-19-172 Small Community Air Service Development

recession and greater consolidation, airlines focused more on closely managing capacity in an effort to control costs and improve profits. Reduced capacity also leads to higher prices, on average.16 Moreover, this focus has led airlines to cut less-profitable routes—which are often to smaller communities—and focus on air service around the country’s largest airports.17 Relatedly, airlines have replaced the 50-seat-or-smaller jets traditionally used to serve small communities with larger, more fuel-efficient 51-to-76-seat aircraft. According to a 2014 MIT study, this shift in aircraft usage can have a negative impact on service to small communities by limiting the potential for profitable air service from these airports.18

To overcome these challenges, small communities and their airports have pursued a variety of strategies to attract or maintain air service. These strategies have involved community support of air service, communication with airline representatives about new service opportunities, and financial incentives for airlines. Such financial incentives include minimum revenue guarantees, which are designed to limit the risk to an airline of initiating air service by guaranteeing it will generate a specified amount of revenue from the ticket sales associated with new service. If the airline does not meet its revenue target, the community makes a payment to the airline to cover the shortfall.

Through SCASDP, communities can apply for federal discretionary grants to fund these and other strategies to improve their air service and address airfare issues at small airports. DOT is authorized to award SCASDP grants to communities with underserved airports that seek to obtain airline service or to implement other measures, including marketing and promotional efforts, to improve the cost and availability of air service.19 The Office of Aviation Analysis in DOT’s Office of the Secretary administers the program, which is funded through FAA’s Airport

16William Spitz, Mitchell O’Connor, Russell Mills, Michael Carroll, and Sonjia Murray, Effects of Airline Industry Changes on Small- and Non-Hub Airports, (The National Academies Press, 2015). 17Michael Wittman and William Swelbar, Trends and Market Forces Shaping Small Community Air Service in the United States, (MIT International Center for Air Transportation, May 2013). 18Wittman, Public Funding of Airport Incentives: The Efficacy of the Small Community Air Service Development Grant Program. 1949 U.S.C. § 41743.

Page 13: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 8 GAO-19-172 Small Community Air Service Development

Improvement Program.20 DOT may award SCASDP grants to a single community or to a consortium of communities in the 50 states, the District of Columbia, Puerto Rico, and U.S. territories and possessions.21

From fiscal years 2002 through 2017, DOT awarded 401 grants totaling approximately $188 million. Individual grants range from $20,000 to nearly $1.6 million. Annual funding for SCASDP has decreased from $20 million in 2002—the first year the program was appropriated funding—to about $5 million in recent years, but was increased to $10 million for fiscal year 2017.22 See figure 1 for an overview of annual grant funds available through fiscal year 2017.23

20The Airport Improvement Program finances capital improvements at U.S. airports. The program is funded through the Airport and Airway Trust Fund, which is funded from excise taxes paid by air passengers. 21A consortium project could involve multiple airports within one state receiving funds to market their service. 22The Federal Aviation Administration Reauthorization Act of 2018 authorizes appropriations of $10 million per year for SCASDP for fiscal years 2018 through 2023. FAA Reauthorization of 2018, Pub. L. No. 115-254, § 455, 132 Stat. 3186 (2018). 23According to DOT officials, the amount of funds available in a given fiscal year has, at times, exceeded the appropriated amount for that year. For example, in certain years funds recovered from prior grant awards became available and supplemented the appropriated funding.

Page 14: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 9 GAO-19-172 Small Community Air Service Development

Figure 1: Available Funding and Number of Grants for the Small Community Air Service Development Program, Fiscal Years 2002–2017

The law establishing SCASDP provides DOT considerable flexibility in implementing the program and selecting projects to be funded, but outlines key statutory requirements for applicants and projects.24 Additionally, DOT has implemented certain eligibility restrictions in its public grant notices over the course of the program, such as restrictions for airport capital improvement projects. See table 2 for an overview of the statutory requirements and other eligibility restrictions in the grant notice.

24Pub. L. No. 106-181, 114 Stat. 92.

Page 15: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 10 GAO-19-172 Small Community Air Service Development

Table 2: Requirements for Applicants and Projects for Small Community Air Service Development Program (SCASDP) Grants, Fiscal Years 2014–2016

Statutory criteria for applicants and projectsa Description Size The airport serving the community or consortium is not larger than a small hub

airport according to FAA hub classifications effective at the time the Office of the Secretary issues a request for proposals, and has • insufficient air service, or • unreasonably high airfares.

Characteristics The airport presents characteristics such as geographic diversity or unique circumstances that will demonstrate the need for, and feasibility of, grant assistance.

State limit No more than 4 communities, consortia, or a combination thereof may be selected from the same state each year.

Overall limit No more than 40 communities or consortia, or a combination thereof, may be selected each year.

Previously funded project A community or consortium may not receive an additional grant to support a project for which it previously received a grant.

Types of assistance • Grants to communities that seek to provide assistance to U.S. airlines to subsidize service for up to 3 years.

• Grants to underserved airports to obtain service or implement other measures appropriate to improve the cost and availability of air service, including through marketing and promotion and enhanced utilization of airport facilities.

Restrictions in the grant noticeb Description Concurrent grants A community or consortium may have only one grant at any time. Airport capital improvement projects Projects including, but not limited to, runway expansions and enhancements,

construction of additional aircraft gates, and other airport terminal expansions and reconfigurations may not receive grants.

Private entities Only public government entities may be legal sponsors of grants.

Source: GAO analysis of 49 U.S.C. § 41743 and Department of Transportation (DOT) SCASDP grant notices. | GAO-19-172

Note: DOT restricted airports participating in the Essential Air Service program from eligibility for SCASDP grants in its fiscal year 2017 grant notice. 82 Fed. Reg. 48575 (Oct.18, 2017). aOriginally, the SCASDP statute required that eligibility be limited to communities that were no larger than small hubs as of 1997, however, this language was amended in a series of appropriations acts to limit eligibility to those communities that were no larger than a small hub as of the date when DOT issued its request for proposals for SCASDP. Finally, a provision in the 2018 FAA Reauthorization Act, Pub. L. No. 115-224, § 455(a), made a permanent change to the provision saying that only airports not larger than small hubs as determined using DOT’s most recently published classification would be considered eligible. b79 Fed. Reg. 38110 (July 3, 2014), 80 Fed. Reg. 35721 (June 22, 2015), and 81 Fed. Reg. 17767 (March 30, 2016).

DOT is directed by statute to give priority consideration to communities or consortia based on certain characteristics. DOT has described these considerations in its grant notices as “priority selection criteria.” DOT has also established a set of “secondary selection criteria” in its grant notices

Page 16: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 11 GAO-19-172 Small Community Air Service Development

to use in its evaluation of applications. See table 3 for a description of both sets of selection criteria.

Table 3: Selection Criteria for Small Community Air Service Development Program (SCASDP) Grants, Fiscal Years 2014–2016

Statutory priority selection criteriaa Description in grant notice Airfares higher than the national average for all communities

Comparison of the local community’s airfares to the national average airfares for all similar markets.

Local contribution of funding A community’s proposal of local funding for the project. Public-private partnership A community’s commitment to facilitate air service in the form of a public-

private partnership. Material benefits Important benefits a project may provide to a broad segment of the

community, such as service that would offer new or additional access to a connecting hub airport, convenient travel times for both business and leisure travelers, and/or lower airfares.

Timely use of assistance A well-defined strategic plan and reasonable timetable for use of grant funds.

Consolidation of air service A consortium effort to consolidate air service into one regional airport. Secondary selection criteria from the grant noticeb Description in grant notice Innovation New and creative solutions to air transportation issues, such as utilizing or

encouraging intermodal or regional solutions to connect passengers to air service.

Community participation Broad community participation, including the community’s demonstrated commitment to and participation in the proposed project.

Location The location and characteristics of a community, including its proximity to large centers of air service and low-fare service alternatives, population and business activity, and whether the community’s proximity to an existing or prior grantee could adversely affect its proposal or the project undertaken by the other grantee.

Other factors Factors including: • whether the proposed project clearly addresses the applicant’s stated

problems; • the community’s existing level of air service and whether it has been

increasing or decreasing; • whether the applicant has a plan to provide financial support for the

project after the requested grant expires; • the grant amount requested compared with total funds available for all

communities; • the grant amount requested compared with the local share offered; • letters of intent from airline planning departments or intermodal surface

transportation providers; • whether the applicant has plans to continue with the proposed project if

it is not self-sustaining after the grant expires; and • equitable and geographic distribution of available funds.

Source: 49 U.S.C. § 41743 and DOT SCASDP grant notices. | GAO-19-172

Page 17: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 12 GAO-19-172 Small Community Air Service Development

Note: DOT also used these selection criteria and descriptions in the fiscal year 2017 grant award cycle. 82 Fed. Reg. 48575 (October 18, 2017). a49 U.S.C. § 41743(c)(5). b79 Fed. Reg. 38110 (July 3, 2014), 80 Fed. Reg. 35721 (June 22, 2015), and 81 Fed. Reg. 17767 (March 30, 2016).

DOT’s evaluation of SCASDP grant applications consists of a multi-phase review process, including: (1) an eligibility review against the eligibility criteria; (2) a technical review of the application’s merit against the priority and secondary selection criteria guided by a written evaluation plan that provides reviewers specific instructions; (3) a control-and-calibration review aimed at ensuring consistency with criteria and program requirements; and (4) a senior team review, which provides recommendations for awards to the Secretary of Transportation, who makes the final decisions. To begin each grant cycle, DOT issues a public notice announcing the grant opportunity and soliciting grant proposals from communities. The grant notices describe the process DOT will use to evaluate and select applications for awards—see figure 2.

Figure 2: The Department of Transportation’s Process for Evaluating Small Community Air Service Development Program Grant Applications, Fiscal Years 2014–2016

Note: DOT provides technical reviewers with the following rating categories for reviewing applications: “highly recommended” applications align with all or most of the selection criteria; “recommended” applications align with most or some of the selection criteria; “acceptable” applications align with some or few; and “not recommended” applications align with very few or none of the selection criteria.

Communities that receive a grant award sign a grant agreement, which details DOT’s requirements and expectations for the grantee. Grants are generally awarded for periods of approximately 2 to 3 years, but DOT

Page 18: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 13 GAO-19-172 Small Community Air Service Development

allows grantees to request extensions based on individual circumstances. SCASDP grants are reimbursable, and therefore communities expend funds for their projects before seeking reimbursement from DOT. Grantees are required to submit quarterly reports to DOT on progress made in implementing their projects and final reports after projects have been completed or grant agreements have expired.25

Since its inception, two studies have sought to assess the effectiveness of SCASDP in helping small communities improve their air service. These studies did so by evaluating the extent to which grantee communities were able to accomplish their project goals.

• In 2008, DOT’s Office of the Inspector General found that 30 percent of fiscal year 2002–2006 grant projects were successful in achieving their grant objectives and sustained the resulting benefits for at least 12 months; 7.5 percent were able to achieve some of their objectives; and 62.5 percent were unable to achieve any of their objectives or were voluntarily terminated prior to making any substantive progress.26

• In 2014, an MIT researcher studied 115 fiscal year 2006–2011 grants and found the success rate ranged from 31 to 43 percent.27 For projects aiming to attract new service, the study defined success as achieving the identified service within 28 months of accepting the grant and maintaining that service throughout the remainder of the 28 months. Grants to market an airport’s existing air service were considered successful if the airport maintained or improved its level of

25For fiscal year 2017 grants, DOT is requiring semi-annual, rather than quarterly, reports from grantees. 26OIG, DOT, The Small Community Air Service Development Program. The study included 40 grants awarded for fiscal year 2002–2006 that had been closed for at least 12 months as of March 31, 2007. The DOT-IG excluded grant projects that were feasibility studies from its scope. 27Wittman, Public Funding of Airport Incentives: The Efficacy of the Small Community Air Service Development Grant Program. The study excluded consortia grants, and any project whose goals were not to obtain new air service or market existing service. DOT officials expressed several concerns with the methodology of this study, including the study’s (1) reliance on publicly available grant information, which does not include information on the full length or final scope of grants; (2) use of the first 28 months of the grant term as the basis for determining whether the project was successful or unsuccessful, even though certain grants were still active; (3) use of frequency of service as a metric to measure marketing grants; and (4) exclusion of certain grants’ alternate goals.

Page 19: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 14 GAO-19-172 Small Community Air Service Development

service for at least 28 months after the grant was awarded. The study discussed factors that may have contributed to unsuccessful projects—such as lack of airline interest in a market or an economic downturn.

We found that certain aspects of DOT’s review of fiscal year 2014–2016 application eligibility and technical merit were inconsistent with the process described in its grant notices, which may have limited applicants’ ability to prepare competitive applications. Specifically, we found that DOT did not evaluate all eligibility requirements identified in the grant notices in the eligibility review phase, and did not apply all selection criteria in the technical review of applications as they were described in the grant notices. DOT officials agreed with our findings and provided us a draft copy of the grant notice for the upcoming grant cycle, which provides better clarity about the SCASDP grant application review process.

We found that DOT evaluated grant applications against some of the eligibility requirements identified in the grant notices in the eligibility review phase and against others in the technical review phase. According to the fiscal year 2014–2016 grant notices, DOT would first determine that applications satisfied all eligibility requirements—and deem applications that did not satisfy these requirements ineligible and remove them from further consideration—before evaluating the merits of eligible applications

DOT’s Award Process Was Inconsistent with Its Grant Notices and Lacked Clear Evaluation Guidance, but Was Consistent with Some Recommended Practices for Awarding Grants DOT’s Process for Evaluating Grant Applications Was Not Fully Consistent with the Process Described in Its Grant Notices

DOT’s Eligibility Review Did Not Include All Eligibility Requirements Identified in the Grant Notices

Page 20: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 15 GAO-19-172 Small Community Air Service Development

against the priority and secondary selection criteria in the technical review.28 Our review shows that DOT evaluated six eligibility requirements in the eligibility review phase, consistent with the process described in its grant notices. DOT applied the requirements that airports larger than a small hub airport, previously funded grant projects, communities with active grants, airport capital improvement projects, private entities, and late applications were not eligible for grants, and excluded 17 of the 142 applications submitted from further consideration. DOT then advanced the remaining 125 applications to the technical review phase.29

However, DOT’s process for evaluating two criteria identified as eligibility requirements in the grant notices—that airports have either insufficient air service or unreasonably high airfares—was not consistent with the process described in the grant notices. We found, and DOT officials acknowledged, that reviewers evaluated these criteria in the technical review phase.30 According to DOT officials, the notices incorrectly listed these criteria as conditions of eligibility, indicating that they would be evaluated before the technical review. DOT officials told us that the SCASDP statute refers to “criteria for participation,” rather than eligibility requirements, and does not specify the phase at which reviewers should apply the criteria. DOT officials told us they reviewed the insufficient air service and unreasonably high airfares criteria in the technical review because officials determined that the technical reviewers have the subject matter expertise to evaluate applications against these criteria—expertise that involves reviewing available data on air service and airfares, conducting market and operational analyses, and considering unique or situational factors that may apply to each applicant.

2881 Fed. Reg. 17767 (Mar. 30, 2016); 80 Fed. Reg. 35721 (June 22, 2015); 79 Fed. Reg. 38110 (July 3, 2014). 29Specifically, one application was submitted in support of a medium hub airport, seven applications proposed projects that were funded in prior years, one application proposed an airport capital improvement project, two applications were submitted by private entities, and six applications were late submissions. 30DOT officials told us they also evaluate the geographic diversity/unique circumstances statutory criterion in conjunction with the priority and secondary selection criteria during the technical review, and in so doing, may look for geographical impediments to air service for a community, such as being situated on an island or surrounded by mountain ranges.

Page 21: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 16 GAO-19-172 Small Community Air Service Development

• Insufficient air service. DOT officials explained that to satisfy the insufficient air service criterion, each eligible applicant must identify in its grant application a specific air service deficiency or need at its airport—such as a major origin/destination market that is not presently served or is not being served adequately—to which its proposed grant project would respond.31 For example, a fiscal year 2014 applicant identified its airport’s lack of nonstop westbound service to a major connecting hub as its air service deficiency. The community sought grant funding for a minimum revenue guarantee to attract American Airlines service to Phoenix-Sky Harbor International Airport as the solution to this deficiency. As part of the technical review phase, reviewers use route maps and air service schedules to verify the facts of the air service deficiency the community identified. DOT determined that each of the 125 applications advanced to the technical review had demonstrated insufficient air service.

• Unreasonably high airfares. DOT officials told us that they use the priority selection criterion that the local community’s airfares are higher than the national average for all communities (or airfare priority selection criterion) to help define this criterion. According to its grant notices, DOT evaluates the airfare priority selection criterion by comparing the airfares on routes from each applicant’s airport to routes throughout the country of similar distance and density to determine the extent to which a community’s fares are above or below the national average. Generally, reviewers described applicants whose airfares were lower than the national average as not aligning with the airfare priority selection criterion, while applicants with higher-than-average airfares were described as aligning with the airfare priority selection criterion.

According to OMB guidance for administering grants, the grant notice should make the application process transparent so that applicants can make informed decisions when preparing applications to maximize fairness of the process. As a result of our audit work, DOT officials told us they plan to revise the grant notice for the upcoming grant cycle to clarify their process for determining eligibility and assessing application merit during the eligibility and technical review phases of the evaluation process. On December 7, 2018, DOT provided us a draft copy of the revised grant notice for the upcoming grant cycle. Based on our review,

31The “Contents of Application” section of the grant notices informed applicants that their applications should include a clear description of their air service needs or deficiencies as well as their present plans or strategies to directly address those needs or deficiencies.

Page 22: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 17 GAO-19-172 Small Community Air Service Development

the draft notice’s description of DOT’s application evaluation process is consistent with the actual process DOT used to evaluate fiscal year 2014–2016 grant applications. Specifically, the draft notice clarifies the restrictions DOT will apply as part of the eligibility review phase, including the restrictions on airports larger than a small hub, previously funded grant projects, communities with active grants, airport capital improvement projects, and private entities. Further, the draft notice clarifies that DOT will evaluate grant applications against criteria for insufficient air service and unreasonably high airfares during the technical review of applications, along with the priority and secondary selection criteria. This revision to the grant notice should improve applicants’ understanding of DOT’s evaluation process and the information needed to make their application competitive. As a result of this action taken by DOT, we are not making a recommendation on this matter in this report.

We found that DOT’s process for evaluating grant applications against the priority selection criteria in the technical review was consistent with the process described in its grant notices, but its process for evaluating the secondary selection criteria differed from the notices. As described in the grant notices, DOT would view applications that aligned well with one or more of the priority selection criteria more favorably than those that aligned with none of the criteria. Consistent with the grant notices, we found that fiscal year 2014–2016 grant applications that reviewers determined aligned with one or more of the priority selection criteria were viewed more favorably by DOT in terms of how highly they were rated. Applications that DOT viewed favorably typically demonstrated evidence of (1) airports with higher-than-average airfares; (2) local community contributions; (3) a public-private partnership, often through the local chamber of commerce and tourism bureau; (4) material benefits to the community due to, for example, the presence of educational facilities, military installations, or tourist-dependent businesses in the area, or new non-stop service for an isolated community; and (5) timely use of the grant assistance, demonstrated through a project timetable, or frequently a letter of support for the project from the airline whose service the community was interested in attracting.

However, we found that DOT’s use of the secondary selection criteria to evaluate applications was inconsistent with the evaluation process described in its grant notices. According to the grant notices, reviewers would use the secondary selection criteria to compare and select among applications that were rated similarly based on the priority selection criteria. In practice, we found that DOT reviewers used the secondary selection criteria when initially rating applications. Additionally, we found

DOT’s Technical Review of Application Merit Did Not Apply All Selection Criteria as Described in the Grant Notices

Page 23: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 18 GAO-19-172 Small Community Air Service Development

that although the grant notices established a distinct definition for each secondary selection criterion, including examples of information DOT would consider relevant to satisfy each criterion, reviewers justified that applications had met priority selection criteria by citing information that, according to the notices, should have only satisfied secondary selection criteria. For example, for 68 applications, reviewers cited a letter of support for an application from an airline (which is defined under the other factors secondary selection criterion) as evidence that the timely use of assistance priority selection criterion had been met. Additionally, for 12 applications, reviewers justified whether or not the material benefits priority selection criterion had been met by citing information about an airport’s existing air service, which is defined under the other factors secondary selection criterion.

According to OMB guidance for administering grants, the grant notice should clearly describe all criteria used to evaluate grant applications. Officials acknowledged that the grant notices did not accurately describe the way in which DOT uses the secondary selection criteria, which in practice has been to help reviewers determine an application’s alignment with the priority selection criteria, not to differentiate between similarly-rated applications. Officials explained that, for example, they regard an airline letter of support for an application as a demonstration of an application’s alignment with both the other factors secondary selection criterion and the timely use of assistance priority selection criterion, although this viewpoint is not reflected in the grant notices. Officials agreed to revise their grant notice to clarify how they use the secondary selection criterion in their evaluation process. We found that DOT’s draft grant notice for the upcoming grant cycle provides a more accurate description of DOT’s use of the secondary selection criteria, which is consistent with the process DOT used to evaluate fiscal year 2014–2016 grant applications. Specifically, the draft grant notice explains that in some cases DOT may use information relating to secondary selection criteria to support a determination of the extent of an application’s alignment with one or more priority selection criteria. This revision to the grant notice should further improve applicants’ understanding of the information needed to make their applications competitive. As a result of this action taken by DOT, we are not making a recommendation on this matter in this report.

Page 24: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 19 GAO-19-172 Small Community Air Service Development

DOT prepared an evaluation plan that included key information needed for reviewers to evaluate application merit in the technical review. According to OMB guidance for administering grants, agencies must design and execute a merit review process. In addition, DOT’s guidance for administering financial assistance awards defines the merit review process as a thorough, consistent, and objective examination of applications based on pre-established criteria. According to DOT’s guidance, agencies should incorporate the merit review process in an evaluation plan, which should (1) describe the process for evaluating applications and identify teams of reviewers; (2) provide reviewers instructions and training; and (3) include information about the criteria used to review and rate applications, documentation of review, and a timeline for the process. We found that the SCASDP evaluation plan provided an overview of the stages of the evaluation process and the teams of reviewers involved at each stage; provided instructions and training for reviewers—including descriptions of the selection criteria included in the grant notices, examples of relevant information to consider for each criterion, and a set of rating categories to reflect the reviewer’s overall assessment of the application—and described the timeline for review.

In reviewing documentation of DOT’s evaluation of grant applications, we found that DOT reviewers followed the process described in the evaluation plan. First, they determined whether an application met each applicable criterion by checking “Yes” or “No” in the applicant evaluation database and recording notes on the information the applicant included in its application relevant to the criteria.32 For example, reviewers recorded the local contribution the community proposed for the local contribution of funding priority selection criterion, and noted the percentage by which its airfares were above or below the national average for the airfares priority selection criterion. Then, reviewers wrote an evaluation narrative for each application in which they cited the various selection criteria the application satisfied, as well as their rating of the application, which was based on the reviewer’s determination of the extent an application aligned with the criteria overall (see table 4).

32Certain selection criteria, such as the regional consolidation of air service priority criterion, are not applicable to every application.

DOT’s Evaluation Plan Provided Reviewers Most Key Information, but Did Not Provide Clear Guidance for Rating Applications

Page 25: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 20 GAO-19-172 Small Community Air Service Development

Table 4: Rating Definitions for Small Community Air Service Development Program (SCASDP) Grant Applications

Rating Description Highly Recommended Project aligns extremely well with the priority and/or

secondary selection criteria (Aligns with “all/most” of the criteria)

Recommended Project aligns well with the priority and/or secondary selection criteria (Aligns with “most/some” of the criteria)

Acceptable Project aligns with the priority and/or secondary selection criteria (Aligns with “some/few” of the criteria)

Not Recommended Project does not align with the priority or secondary criteria (Aligns with “very few/none” of the criteria)

Source: Department of Transportation’s SCASDP guidance. | GAO-19-172

However, we found that the evaluation plan did not provide reviewers clear guidance regarding assigning these ratings based on an application’s alignment with the selection criteria. Specifically, the evaluation plan outlined relevant information the reviewer should consider for each criterion, but only described how reviewers should consider an application’s alignment with individual criteria for the local contribution of funding priority criterion.33 Reviewers were therefore left to determine what constituted alignment for the remaining selection criteria. For example, the evaluation plan indicates that reviewers should consider a community’s businesses, educational institutions, and local attractions for the material benefits priority selection criterion, but it does not clarify how an application would align better or worse with this criterion. Further, the evaluation plan did not state how many of the selection criteria an application must align with to receive a “highly recommended,” “recommended,” “acceptable,” or “not recommended” rating, and the rating categories were not clearly delineated from one another. Specifically, the definitions for three of the four rating categories overlapped with the definition of another rating category, introducing more reviewer discretion into the review process. Based on the rating category definitions (see table 4), an application that reviewers determined aligned with “most” of the selection criteria could qualify as either “highly recommended” or “recommended,” while an application that was

33Specifically, the evaluation plan states that applications providing proportionately higher levels of cash contributions from sources other than airport revenues “illustrate deeper alignment, and multiple levels of funding illustrate deeper alignment in terms of strength of alignment.”

Page 26: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 21 GAO-19-172 Small Community Air Service Development

determined to align with “some” of the selection criteria could be rated “recommended” or “acceptable” at this stage.

DOT officials explained that their ratings are based on examining each application against the selection criteria holistically in terms of both breadth (the number of selection criteria an application aligns with) and depth (the degree to which an application aligns with any individual selection criterion). They explained that reviewers do not assess applications against each selection criterion in a vacuum, nor do they assign specific weights or values to each individual criterion individually. Officials told us that they consider the “whole picture” in terms of how a project aligns with the criteria and rate it accordingly. This approach, however, is inconsistent with DOT’s financial award guidance, which calls for a clearly defined application evaluation process in order for reviewers to apply the rating system fairly and accurately. Because the evaluation plan did not clarify how reviewers should determine an application’s alignment with the selection criteria, and application ratings were based on these determinations, the evaluation plan did not provide assurance that the basis for DOT’s ratings would be consistent across applications.

As a result of our audit work, DOT officials told us they intend to update the evaluation plan for the upcoming grant cycle to provide additional examples of how reviewers should determine alignment for each criterion and agreed to look into ways to provide greater differentiation between the rating categories. However, as of March 2019, neither the exact changes DOT officials intend to make to the evaluation plan or a time frame for making them have been determined. Without providing reviewers with additional guidance in the evaluation plan on how to determine alignment with the selection criteria and rate applications, DOT lacks assurance that applications will be consistently reviewed.

In previous reviews of discretionary grant programs, we identified recommended practices for awarding federal discretionary grants based on our review of grant-making policies and guidance used by OMB and four other federal agencies, including DOT.34 Following these practices can help ensure discretionary grants are awarded using a fair and objective evaluation process. We found that DOT’s grant award process

34The other agencies were the Departments of Commerce, Education, and Labor. See GAO-11-283.

DOT Followed Certain Recommended Practices for Awarding Discretionary Grants and Partially Followed Others

Page 27: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 22 GAO-19-172 Small Community Air Service Development

fully followed two and partially followed three recommended practices for awarding discretionary grants (see table 5).

Table 5: GAO’s Assessment of the Extent to Which the Small Community Air Service Development Program (SCASDP) Followed Recommended Practices for Awarding Discretionary Grants (Fiscal Years 2014–2016)

Practice Attributes of practice Extent followed Communicate with potential applicants prior to the competition

Provide information prior to making award decisions on available funding, key dates, competition rules (i.e., eligibility, technical review of application, and selection criteria), funding priorities, types of projects to be funded, outreach efforts to new applicants and pre-application assistance.

Partially

Develop a technical review panel with certain characteristics

Use a technical review panel consisting of reviewers who hold relevant expertise, do not have conflicts of interest, apply the appropriate criteria, and are trained.

Fully

Plan for administering the technical review Develop a plan for the technical review that describes the number of panels and reviewers and includes methods for assigning applications to review panels, identifying reviewers, recording the results of the technical review, resolving scoring variances across panels, and overseeing the panel to ensure a consistent review.

Partially

Notify applicants of awards decisions Notify unsuccessful and successful applicants of selection decisions in writing and provide feedback on applications.

Partially

Document rationale for awards decisions Document the rationale for awards decisions, including the reasons individual projects were selected or not selected and how changes made to requested funding amounts may affect applicants’ ability to achieve project goals.

Fully

Source: GAO analysis of federal agency guidance and the Department of Transportation’s (DOT) SCASDP evaluation and selection process. | GAO-19-172

Notes: We assessed the extent to which DOT followed each practice using the following ratings: “Fully”: Agency addressed all attributes of the practice; “Partially”: Agency addressed some, but not all attributes of the practice; and “Not Followed”: Agency did not address any attributes of the practice.

Previous GAO work also identified a practice to “assess applicants’ ability to account for funds,” a concept that relates to determining if applicants met eligibility requirements, checking previous grant history, assessing financial management systems, and analyzing project budgets. We did not assess this practice in our evaluation because SCASDP grants are disbursed on a reimbursable basis; therefore, communities make the initial expenditures for projects and then seek reimbursement from DOT.

We compared DOT’s process with the following five recommended practices.

• Communicating with potential applicants prior to the competition: Partially followed. DOT communicated information

Page 28: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 23 GAO-19-172 Small Community Air Service Development

about SCASDP grants—including available funding, key dates, eligibility and selection criteria, funding priorities, and types of projects to be funded—primarily through its grant notices. However, as discussed above, information in the grant notices regarding eligibility and DOT’s use of the secondary selection criteria was not entirely consistent with DOT’s actual process for evaluating fiscal year 2014–2016 grant applications. In addition, to help potential applicants understand application requirements, DOT provided pre-application technical assistance to communities upon request.

• Develop a technical review panel with certain characteristics: Fully followed. DOT assembled technical review teams that were comprised primarily of analysts from the Office of Aviation Analysis. Reviewers were given training slides and guidance on the grant notice. Further, DOT conducted annual training sessions for reviewers, and team leaders worked with team members to make sure that the material was clear. Although DOT review team members were not required to submit specific conflict-of-interest forms for SCASDP, according to DOT officials, all new DOT personnel undergo ethics orientation, including pertaining to potential conflicts of interest.

• Plan for administering the technical review: Partially followed. DOT developed an evaluation plan that outlined the eligibility, technical review, and control-and-calibration teams’ roles, how applications would be assigned for review, and how results should be recorded in a centralized database. The evaluation plan outlined the priority and secondary selection criteria defined in statute and grant notices, and provided examples of application information that should be considered relevant to each criterion. However, as discussed previously, the evaluation plan did not provide clear guidance to reviewers about determining an application’s alignment with each criterion, and its rating categories were not clearly delineated. Without such guidance, reviewers might not review and rate grant applications consistently.

• Notify applicants of award decisions: Partially followed. DOT posted annual written grant award orders on its public docket and its website that identified successful fiscal year 2014–2016 applicants. Further, DOT’s award order for fiscal year 2017 grants states that DOT will from thereon provide the order to all applicants. According to DOT officials and several fiscal year 2014–2016 applicants we spoke with, DOT provided unsuccessful applicants feedback on their applications upon request. However, officials told us they did not consider the need to provide explicit language in the grant notice or the award order that this opportunity existed. We spoke with airport

Page 29: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 24 GAO-19-172 Small Community Air Service Development

officials from six communities that applied unsuccessfully for grants during the fiscal 2014–2016 grant period. Of those, three communities had not reached out to DOT and were unaware of the opportunity to do so. The remaining three communities solicited feedback from DOT, but only two received a response, according to airport officials we interviewed. DOT officials told us they were unable to find any record of a request for feedback from the third community. We have previously reported that timely and substantive feedback can help unsuccessful applicants determine whether to expend resources to apply in future rounds.35 Because DOT did not make applicants aware that it was willing to provide feedback, some unsuccessful applicants may not have received the information they needed to revise their proposed project or application content to better compete in future rounds. In response to our work, DOT’s draft grant notice for the upcoming grant cycle includes language that applicants may request feedback on their application from DOT. This change should help make applicants aware of the opportunity for feedback and, in particular, could help unsuccessful applicants determine whether to apply again and, if so, how to improve their applications in future grant cycles.

• Document the rationale for awards decisions: Fully followed. DOT reviewers entered their evaluation of each grant application against the selection criteria into a centralized database and provided a written narrative justification that described the grant project, outlined the selection criteria addressed, and the degree to which the project aligned with the criteria. Finally, DOT awarded successful applicants the entire grant amount requested in nearly all instances in the fiscal year 2014–2016 grant cycles; for two applicants, DOT did not document its reason for not awarding the entire amount requested or how the reduced award might affect their ability to achieve project goals.

35GAO, Discretionary Transportation Grants: DOT Should Take Actions to Improve the Selection of Freight and Highway Projects, GAO-18-38 (Washington, D.C.: Nov. 2, 2017).

Page 30: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 25 GAO-19-172 Small Community Air Service Development

DOT monitors the progress of grant projects by reviewing periodic reports submitted by grantees and, at their conclusion, collects information on projects’ effectiveness. Specifically, DOT required fiscal year 2010–2014 grantees to submit quarterly reports to DOT that included a brief narrative detailing the status of the project and its progress, and status updates on the hiring of any consultants in support of the project, progress toward completion of the community’s in-kind contributions, marketing or promotional activities, and contract negotiations with airlines or any other third parties related to the project’s implementation.36 DOT officials told us they use the information on project status in the quarterly reports primarily as a means of corroborating information grantees submit in their requests for reimbursement. Also, they use data from the Bureau of Transportation Statistics—such as scheduled departures to monitor project outcomes such as an airport’s efforts to attract new air service.

Grantees were also required to provide final reports within 3 months of the project’s conclusion or the grant agreement’s expiration, which, according to DOT officials, have led to improvements to program administration. The final report consists of a questionnaire for the grantee to complete on topics including project results—such as whether projects resulted in an increase in passenger enplanements, a reduction in the number of passengers choosing to use alternative airports, lower airfares, or increases in capacity—and the adequacy of the federal funds awarded for the project. The final report also includes a section for grantees to rate

36According to DOT’s grant notice, in-kind contributions from the community may include items such as donated advertising from media outlets or catering services for inaugural events. Travel banks and travel commitments or pledges are also considered to be in-kind contributions.

DOT Collects Information to Oversee Grant Projects and Monitors Grantee Performance in Establishing or Expanding Air Service DOT Uses Quarterly and Final Grantee Reports to Oversee Projects and to Improve Grant Administration

Page 31: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 26 GAO-19-172 Small Community Air Service Development

SCASDP on, among other things, the ease and clarity of the application process, grant agreement, and reporting and reimbursement requirements; the program’s effectiveness in addressing air service and fare issues; and overall satisfaction with the program.

Finally, grantees have the option to provide a written narrative assessing the success or failure of their projects and offer any suggestions on improvements to the program. DOT officials told us that based on information provided by grantees in final reports as well as oral feedback from grantees, they have made a number of improvements to program administration, including making the quarterly reporting requirement semi-annual for the fiscal year 2017 grant cycle, modifying the grant application format, and clarifying information provided in the grant notice, such as the restriction on projects that previously received grants and how communities may use grants for marketing programs. According to officials, information provided in final reports has not led them to make larger changes to the program, because in certain years airline industry dynamics—such as the financial losses mainline airlines suffered after the financial crisis and recession from 2007–2009—may have played more of a role in the outcomes of SCASDP projects than the individual circumstances of each grant project.

DOT has monitored grant performance since 2012 through a database to track air service outcomes for all grants. DOT officials explained that they track whether the grant helped the airport establish air service, the longevity of service and any cessations, and reasons service was terminated—such as a bankruptcy, merger, or pilot shortage—based on information included in a grantee’s final report or in the public domain. The officials also noted that they monitor in perpetuity air service levels at all airports that have been awarded grants. Officials told us that, upon request, they have provided information to OMB on the number of grants awarded to communities seeking to establish air service, the goals of the grant projects, and the length of time the service existed. DOT officials told us that they have also conducted internal assessments of closed grants to examine project outcomes in recent years. For example, based on cases in which several fiscal year 2009, 2010, and 2011 grantee communities were able to secure new air service after several grant extensions provided by DOT, these assessments indicated to DOT that longer grant terms were warranted. As a result, DOT revised the fiscal year 2017 award order to extend the length of grant terms and to provide an expedited procedure for initial grant extensions that demonstrate merit. Officials told us that they monitor the performance of grants that are not

DOT Continually Monitors Air Service Trends at Grantee Airports

Page 32: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 27 GAO-19-172 Small Community Air Service Development

seeking new or expanded service, such as those awarded to conduct studies or promote an airport’s existing service, through the aforementioned quarterly and final reports until the grants are formally closed.37

DOT officials explained that they have not established performance measures beyond monitoring air service gains because of limited resources as well as the difficulty of defining what would constitute a successful grant project in a challenging industry environment for small community airports. In particular, officials explained that, in the current industry environment, retaining existing air service may be a more realistic expectation for certain small airports than trying to attract additional air service through a minimum revenue guarantee.

Our analysis of 66 closed fiscal year 2010–2014 grant projects—those projects that had been completed or whose grant agreements had expired—found that half were successful in achieving their grant goals during the grant award period, and over one-third sustained their results for at least 24 months after their grant ended. The success of grant projects, according to selected grantee airports and stakeholders in small community air service issues, is affected by a variety of community, airport, and airline factors, such as the quality and reliability of an airline’s service. Participation in SCASDP also brought additional benefits to communities, including increased connectivity for almost half of grantees and, according to grantee airport officials, increased airport revenues, increased community awareness of the airport, and additional service.

37Additionally, DOT officials stated that they fully monitor in perpetuity any grant that establishes air service, regardless of the stated purpose of the grant. For example, several marketing grants have created air service, and thus are measured in perpetuity by DOT.

Half of Fiscal Year 2010–2014 Grant Projects Were Successful in Achieving Their Goals, Which Selected Airports and Stakeholders Attributed to Several Factors

Page 33: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 28 GAO-19-172 Small Community Air Service Development

In fiscal years 2010 to 2014, DOT awarded 122 grants across 45 states and 2 U.S. territories. The large majority of grants were awarded to individual airports, four grants were awarded to consortia of airports, and four airports won multiple grants within that timeframe (see fig. 3). Of the 122 grants awarded, our analysis included data associated with 66 closed grants as of December 31, 2017.38

Figure 3: Airports That Received Small Community Air Service Development Program Grants, Fiscal Years 2010–2014

We determined that 33 of 66 projects with goals to establish, expand, or promote the small airport’s air service were successful, meaning that they were able to (1) gain new airline service that did not exist prior to 38We initially identified a population of 75 fiscal year 2010–2014 grants awarded by DOT that had been closed or had expired as of December 31, 2017. We subsequently excluded 9 grants from this analysis. See appendix I for more information on our methodology.

Half of Closed Fiscal Year 2010–2014 Grants Met Their Project Goals, and Over One-Third Sustained Improvements

Page 34: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 29 GAO-19-172 Small Community Air Service Development

receiving the grant; (2) expand existing air service from an airline—e.g., by adding flights or increasing flight frequencies; or (3) increase the number of passengers using the community airport through marketing. We categorized grants that fell into more than one of these categories—or that involved multiple airports as part of a consortium—as multi-goal grants. Of the remaining grants, we found that 16 grants were partially successful and that 17 grants were unsuccessful in achieving their stated goals. See table 6.

Table 6: Evaluation Results for Closed Small Community Air Service Development Program (SCASDP) Grants, Fiscal Years 2010-2014

Grant project type

Number of grant projects

Unsuccessful grantsa

Partially successful

grantsb Successful

grantsc

12-month sustained

success post-grant

24-month sustained

success post-grant

Establish Service 37 9 6 22 19 18 Promote Service 21 7 4 10 7 5 Expand Service 2 1 1 0 not applicable not applicable Multi-Goal 6 0 5 1 1 1 Total 66 17 16 33 27 24

Source: GAO analysis of SCASDP data. | GAO-19-172 aUnsuccessful grants were not able to accomplish their goals at any point within the grant period. bPartially successful grants were able to partially accomplish their goals within the grant period. cSuccessful grants were able to fully accomplish their goals within the grant period.

As shown in table 6, we determined whether the 33 successful projects were able to sustain their success for periods of 12 or 24 months after their grant ended. For 27 of the 33 successful grantees, the outcomes of their grant continued for at least 12 months after the grant ended.39 Twenty-four of these projects sustained their outcomes at least 24 months after the grant ended.

We found that grant projects aiming to establish new air service were the most likely to be successful and sustain their outcomes after the grant ended. Specifically, of the 37 grants to establish service, over half (22) 39We were unable to determine whether two grantees had sustained their success for 12 and 24 months after their grants ended. Specifically, data were not available to evaluate one successful project’s air service 12 months after the grant ended. We did not evaluate the second project at 24 months post-grant because the airport’s EAS destination—the basis for its grant project—changed between 12 and 24 months after its grant ended.

Page 35: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 30 GAO-19-172 Small Community Air Service Development

were successful in terms of maintaining the service at the end of their grant period, and 18 of those grants kept the service for 24 months after the grant ended. Comparatively, less than half of 21 projects to promote service were successful (10), and only one-quarter maintained this success 24 months after the grant ended (5).

Through interviews with 24 grantee airports and 13 stakeholders in small community air service, and a correlation analysis, we found that a variety of factors are associated with the success of grant projects.40 Grantee airports and stakeholders identified eight distinct factors: (1) community demand for air service, (2) a strong or growing local economy, (3) community support for a project, (4) airport operating costs, (5) a grantee airport’s proximity to other airports, (6) an airport’s current service levels, (7) the quality of service provided by an airline, and (8) an airline’s support for a project. We categorized these factors into three groups: community, airport, and airline.

Stakeholders identified community attributes such as demand for air service, a strong or growing local economy, and community support for a project as key factors in the outcomes of projects. Officials from eight grantee airports and three stakeholders we interviewed told us that demand for air service from the community is a factor in successful projects. For example, one grantee said its project successfully established service because there was high demand for eastbound connecting flights from the community. Relatedly, officials from two grantee airports and three stakeholders said a strong or growing local economy, evidenced by high disposable incomes and new businesses, can contribute to a project’s success. One grantee noted that the arrival of a new oil and gas company and the jobs it created provided residents income to use for travel and led to an increase in enplanements at the local airport. The extent of community support for the SCASDP project in the form of funding and collaboration with the airport is also an important indicator of project success, according to five grantee airports and five stakeholders. For example, one aviation consultant said that an engaged business community translates to more purchased tickets and more successful service. Alternately, one grantee airport official stated that the

40The 13 stakeholders in small community air service included four airlines, four airline consultants, two industry trade associations, and three academic experts/researchers.

Grantee Airports and Aviation Stakeholders Attributed Grant Project Outcomes to a Variety of Community, Airport, and Airline Factors

Community Factors

Page 36: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 31 GAO-19-172 Small Community Air Service Development

community’s lack of support led to the withdrawal of the community’s portion of project funding, which led to the grant project’s failure.

Stakeholders also noted that the local airport’s operating costs, distance to other airports, and current levels of service can affect the success of a grant project, according to officials from six airports and seven stakeholders. One grantee airport and three stakeholders said that low operating costs can make an airport more attractive to airlines considering new service. Additionally, a grantee airport that is within a reasonable driving distance of other airports may have less likelihood of success, according to one grantee airport and three stakeholders, as the community has other options for air service besides the local airport. One airline official told us that if a potential passenger can drive to a larger airport with lower fares, they will not pay the premium fare to fly out of their local airport. Officials from four grantee airports and one stakeholder told us that an airport’s current levels of service can contribute to success, as airports that already have scheduled service have proven the viability of their market and can more easily persuade airlines to begin additional service. For example, one grantee said that the success of its service to Atlanta initiated by a SCASDP grant was a catalyst in another airline beginning service to Charlotte.

The quality and reliability of an airline’s service and an airline’s decision to provide a letter of support for a project affects its potential for success. Officials from two grantee airports and one stakeholder stated that the quality of service, in terms of timeliness and frequency of flights, can influence the outcome of projects. For example, the timing of one community’s grant-initiated service to New York allowed for day trips, which were attractive to the community, and the support of the service made the project a success. Two grantees—one whose project was partially successful and one whose project was unsuccessful—also stated that poor service quality led to their project outcomes. An airline’s support for a project is typically communicated through a letter to DOT supporting the grant application. Representatives from three airlines we spoke with had differing views on providing letters of support for grant applications. A representative from one airline told us that letters of support are no longer provided to grant applicants based on the airline’s experience with a community that publicized that the airline was initiating service before the airline had committed to doing so. Representatives from the remaining two airlines noted they provide letters when they have committed to serving the market or when they believe service will start within a few years.

Airport Factors

Airline Factors

Page 37: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 32 GAO-19-172 Small Community Air Service Development

We operationalized measures for 6 of the 8 factors identified by grantee airports and aviation stakeholders with available data (see table 7), and conducted a correlation analysis to determine the extent to which any individual factor correlated with successful outcomes for projects that aimed to establish or promote service.41

Table 7: Factors Affecting the Success of Small Community Air Service Development Projects and Measures

Factor Measure Community demand for air service Population within a 60-minute drive of the airport Strong/growing local economy Income per capita and establishments within a 60-

minute drive of the airport Community support for project Community’s cash contribution Distance to other airports Distance from each grantee airport to the nearest

medium or large hub airports Current service at the airport Grantee’s status as a small hub or Essential Air

Service airport Airline support for project Airline letter of support

Source: GAO analysis. | GAO-19-172

We found that no single factor strongly correlated with project success. However, community income per capita had the strongest positive correlation with success for projects to establish new air service (0.39). For projects promoting the airport’s service, Essential Air Service (EAS) status had the strongest—albeit negative—association with success (-0.33), suggesting that EAS airports are less likely than other grantees to have success in increasing passenger enplanements through SCASDP grants. The remaining factors—including being a small hub airport, having an airline letter of support for the project, and the number of establishments, such as commercial businesses, in the surrounding community—had smaller but positive levels of association with success, 41While the characteristics associated with project outcomes may be best understood using an analysis that considers multiple factors at once, due to the limited number of cases such an analysis was not possible. Instead we used a correlation analysis that measures the strength and direction of the linear relationship between two variables without controlling for the effects of other characteristics. Coefficient variables take a value between negative 1 and 1. The magnitude of the correlation coefficient determines the strength of the correlation. A correlation coefficient of zero would indicate that there was no relationship between the variables. A correlation coefficient close to 1 would indicate a strong positive relationship, while a correlation coefficient close to negative 1 would indicate a strong negative relationship.

Page 38: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 33 GAO-19-172 Small Community Air Service Development

findings that suggest that these factors are not, in isolation, primary drivers of success for projects aiming to establish or promote service.

We found that grantees received benefits from participation in SCASDP beyond the degree to which they accomplished their project goals. Specifically, based on our analysis of 60 grants, we found that 26 grantees experienced an increase in their airport’s connection to the national air transportation system during their grant periods, and officials from grantee airports we interviewed identified additional benefits from participation in SCASDP.42

We evaluated how an airport’s overall connections to the air transportation system—or, connectivity—changed during the grant period. We used a modified version of the International Air Transport Association’s measure of connectivity, which provides an indication of the destinations served from an airport, the frequency of service to each destination, and the number of connections available from each destination.43 An airport’s connectivity may increase if it either experiences an increase in available seats or its destination airport experiences an increase in enplanements, which the measure uses as an indication of connections available through that airport.44 For example, see figure 4 for one grantee airport’s change in connectivity during the grant period. By comparing an airport’s percent change in connectivity (from the year prior to the beginning of the grant to the last year of the grant) to the percent change in a comparison group of similarly-sized

42For our analysis of airport connectivity changes, we excluded 15 grants from the population of 75 fiscal year 2010–2014 grants awarded by DOT that had been closed or expired as of December 31, 2017. See appendix I for more information on our methodology. 43IATA’s analysis uses a connectivity measure based on the number of available seats to each destination served for the first week in July in each year between 1996 and 2005. We modified IATA’s measure to include available seats per calendar year rather than per week in our period of analysis. 44Smyth, M., & Pearce, B. (2007). Aviation Economic Benefits, IATA Economic Briefings No 8.

Data Indicate That Close to Half of Fiscal Year 2010–2014 Projects Increased Their Airport’s Connectivity, and Grantees Identified Additional Benefits from Grant Participation Connectivity

Page 39: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 34 GAO-19-172 Small Community Air Service Development

airports, we were able to determine whether an airport experienced a positive change in its connectivity during its grant period.45

45For the purposes of this report, “Increased connectivity” refers to a grantee airport’s positive percent change in average connectivity that is also 5 percentage points or more higher than its comparison group’s change in average connectivity.

Page 40: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 35 GAO-19-172 Small Community Air Service Development

Figure 4: Comparison of Connectivity Change for a Fiscal Year 2011 Small Community Air Service Development Program Grantee

Note: An airport’s weight is a proxy for the extent of additional connectivity it provides as a destination.

Page 41: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 36 GAO-19-172 Small Community Air Service Development

We identified a total of 26 of 60 grantees that experienced increased connectivity compared to their comparison airports. This total included 17 of the 30 grantees that successfully met their project goals and 8 grantees that were unsuccessful or partially successful in meeting their goals. Similar rates of connectivity increases occurred across goal categories. Specifically:

• Establish Service: Fifteen of 32 grantees that sought to establish service increased their connectivity compared to their comparison group, including 12 of 20 successful projects, 2 partially successful projects, and notably 1 unsuccessful project. Specifically, the fiscal year 2010 grantee community that was unsuccessful in attracting new service nevertheless saw a connectivity increase due to a rise in available seats to Hartfield-Jackson Atlanta International—currently the world’s busiest airport—over the grant period, compared to the connectivity decline experienced in its comparison group. Alternately, 8 projects whose goals were to establish service were successful but did not see an increase in connectivity. For instance, one fiscal year 2011 grantee obtained the service to Denver International Airport that was targeted in its grant application, but experienced an overall decline in connectivity due to a loss of available seats from its airport to other destinations such as Newark Liberty International Airport and Orlando International Airport from 2010 to 2014, the year its grant ended. In another example, a fiscal year 2012 grantee was able to establish service to Orlando Sanford International Airport; however, because this airport is relatively small in terms of passenger enplanements, its additional seats added little to the airport’s connectivity measure. Additionally, available seats to Chicago O’Hare International Airport, a heavily weighted airport, declined over the grant period, leading to an overall decrease in its connectivity.

• Promote Service: Seven of the 20 grantees with projects to promote service increased their connectivity compared to their comparison group, including 4 of 9 successful projects, 1 partially successful project, and 2 unsuccessful projects. For example, one fiscal year 2011 grantee was successful in increasing its passenger enplanements, but did not see an increase in connectivity relative to its comparison group of 11 airports. Additionally, among the two projects that were unsuccessful, one fiscal year 2012 grantee experienced a connectivity increase unrelated to its SCASDP participation because it lost unsubsidized service to Minneapolis-St. Paul International Airport and began receiving EAS service to the larger Chicago O’Hare during our study period.

Page 42: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 37 GAO-19-172 Small Community Air Service Development

• Expand Service, Multi-Goal, and Other: Four of the 8 projects with expand service goals, multiple goals, or other goals increased their connectivity compared to their comparison group, including 1 successful project, 1 partially successful project, and 1 unsuccessful project.46

Airport officials we interviewed identified other ancillary benefits to participation in SCASDP. Officials from four airports stated that the marketing or new service funded by the grant brought additional revenues to their airport. For example, officials from two airports said their enplanements levels reached the 10,000-enplanements-per-year threshold required to receive $1 million in Airport Improvement Program grants. These grant funds can be used for eligible projects, generally those that enhance capacity, safety, or environmental concerns such as runway construction and rehabilitation, airfield lighting, and airplane noise mitigation.47 Officials from seven airports stated their participation informed airlines about their community, leading to some receiving additional service that was not funded by the grant. For example, according to one airport’s marketing manager, its successful fiscal year 2012 grant allowed the airport to show that it was a viable market, and according to the manager, the airport subsequently experienced a 25 percent increase in capacity over the prior 5 years through new and expanded service. Finally, officials from five airports noted that the grant raised the community’s awareness of the local airport. For example, one airport official noted that receiving a federal grant was headline news in the local media.

SCASDP grants can play an important role in helping small communities address challenges they face in developing air service. Given the demand for these grants, it is critical that DOT’s process for evaluating applications and awarding grants is transparent and consistent with the evaluation and award process described in its grant notice. DOT acknowledged the inconsistencies we identified in its evaluation process relative to the grant notice and, in its draft copy of the revised grant notice for the upcoming grant cycle, clarified the description of the eligibility and

46This does not add up to four because one of the grantees that experienced increased connectivity was categorized as an “Other” project which we did not include in our analysis of grant project success. 4749 U.S.C. §§ 47102 and 47114.

Other Benefits

Conclusions

Page 43: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 38 GAO-19-172 Small Community Air Service Development

technical review phases to match its actual evaluation process. Also, the revised notice included language to make applicants aware of the opportunity to request feedback, which could help unsuccessful applicants determine whether to apply again, and if so, how to improve their applications in future grant cycles. These steps by DOT to revise its grant notice should provide applicants greater transparency into the grant-making process. With regard to evaluating applications, DOT’s evaluation plan includes several pieces of key information needed for reviewers to evaluate application merit in the technical review. However, the plan lacks clarity in how reviewers should evaluate and rate applications. Without additional clarity about how reviewers should determine an application’s alignment with the selection criteria, DOT lacks assurance applications are consistently evaluated and rated.

We are making the following recommendation to DOT:

The Secretary of Transportation should clarify in the SCASDP evaluation plan how reviewers should assess a grant application’s alignment with the priority and secondary selection criteria and assign the application rating categories. (Recommendation 1)

We provided a draft of this report to DOT for review and comment. In its official comments, reproduced in appendix III, DOT agreed with our recommendation and provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees and the Secretary of Transportation. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

Recommendation for Executive Action

Agency Comments

Page 44: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Page 39 GAO-19-172 Small Community Air Service Development

If you or members of your staff have questions about this report, please contact me at (202) 512-2834 or [email protected]. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Major contributors to this report are listed in appendix IV.

Heather Krause Director, Physical Infrastructure Issues

Page 45: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 40 GAO-19-172 Small Community Air Service Development

This report (1) examines the extent to which the Department of Transportation’s (DOT) process for awarding Small Community Air Service Development Program (SCASDP) grants is consistent with its grant notices, internal evaluation plan, and recommended practices for discretionary grant programs; (2) identifies steps DOT has taken to oversee grant projects and monitor their performance; and (3) examines the extent to which fiscal year 2010–2014 SCASDP grants assisted airports in improving their air service, and identifies factors affecting the success of grant projects.

To examine the extent to which DOT’s grant award process is consistent with its grant notices and evaluation plan, we first identified DOT’s process for awarding fiscal year 2014–2016 grants through a review of documentation including the three public notices for these grants, DOT’s evaluation of fiscal year 2014–2016 grant applications, award orders announcing fiscal year 2014–2016 grantees, and statements made by DOT officials regarding implementation of the grant program.1 We evaluated DOT’s award process for fiscal year 2014–2016 grants because they were the three most recent award cycles completed when we began our audit work. We then compared DOT’s actual award process to the process for evaluating grant applications for eligibility and technical merit described in DOT’s grant notices for fiscal years 2014–2016 and the evaluation plan DOT used to guide the evaluation of grant applications. We interviewed DOT officials responsible for administering the grant award process. We also compared DOT’s award process to federal guidance from the Office of Management and Budget (OMB) and DOT for administering grants.2

We obtained the perspectives of grant applicants on DOT’s award process by speaking with a non-generalizable sample of 12 applicants for fiscal year 2014, 2015, and 2016 grants—six applicants that were awarded grants and six that were not—on their experiences with the SCASDP application process and DOT’s communication during and after the grant application and award process. The successful applicants we selected reflect a mix of grant years, airport hub sizes, types of projects, applicants that were prior grantees, and previously unsuccessful applicants. The unsuccessful applicants we selected reflect a mix of grant

181 Fed. Reg. 17767 (Mar. 30, 2016); 80 Fed. Reg. 35721 (June 22, 2015); 79 Fed. Reg. 38110 (July 3, 2014). 22 C.F.R. Part 200. DOT, Financial Assistance Guidance Manual (December 2016).

Appendix I: Objectives, Scope, and Methodology

Page 46: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 41 GAO-19-172 Small Community Air Service Development

years, airport hub sizes, and application ratings that were similar to applicants that were awarded grants. Because this was not a random or representative sample, the views of these applicants cannot be generalized to all SCASDP applicants. We also interviewed representatives from four aviation consulting firms to obtain their perspectives on DOT’s award process and learn how they assist communities with compiling applications and executing projects. We selected these firms because they provided assistance to grantees within our scope of work. See appendix II for a complete list of the aviation industry stakeholders we interviewed.

To determine the extent to which DOT’s process for awarding grants is consistent with recommended practices for awarding discretionary grants, we compared the process to these practices. Specifically, the practices relate to (1) communicating with potential applicants prior to the competition, (2) planning for administering the review of applications, (3) developing a technical review panel with certain characteristics, (4) assessing applicants’ abilities to manage grant funds, (5) notifying applicants of decisions, and (6) documenting reasons for award decisions.3 Because SCASDP grants are disbursed on a reimbursable basis, and therefore, communities make initial expenditures for projects prior to seeking reimbursement from DOT, we did not apply the practice related to assessing applicants’ ability to account for funds in our evaluation. To determine the extent to which DOT followed these recommended practices for awarding discretionary grants, we compared their attributes to information from the fiscal year 2014–2016 grant notices, the evaluation plan DOT uses to guide the merit review of grant applications, DOT’s evaluation spreadsheets for fiscal year 2014–2016 grant applications, award orders announcing fiscal year 2014–2016 grantees, and statements made by DOT officials regarding implementation of the grant program. For this effort, one analyst carried out the comparison and a second analyst verified the comparison results. Where differences existed, the analysts discussed them and reached agreement.

To identify the steps DOT has taken to oversee grant projects and monitor their performance, we obtained and reviewed a sample of quarterly project status reports as determined by DOT officials and final reports submitted to DOT by fiscal year 2010–2014 grantees. We also

3GAO-11-283.

Page 47: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 42 GAO-19-172 Small Community Air Service Development

interviewed DOT program officials on their processes for monitoring and evaluating grant projects. Additionally, we interviewed airport officials from or consultants working with 20 fiscal year 2010–2014 grantee communities, selected using a stratified, random sample, about their communication with DOT regarding project progress. The sample includes 10 airports whose projects we determined were successful, 5 airports whose projects we determined were partially successful, and 5 airports whose projects we determined were unsuccessful.

To determine the extent to which fiscal year 2010–2014 grants assisted airports in improving their air service, we evaluated the extent to which grantees were successful in accomplishing their individual project goals. We evaluated fiscal year 2010–2014 grant projects because they represented the most recent grant cycles for which the majority of projects had been completed when we began our audit work. In order to ensure a consistent unit of analysis, we identified a population of 75 fiscal year 2010–2014 grants awarded by DOT that were closed or had expired as of December 31, 2017.

To identify each grantee’s project goal(s), we collected project documentation including grant applications, grant agreements, closeout letters, and final reports submitted to DOT. We reviewed each grantee’s project documentation to identify each project’s principal goal(s). We excluded any project goals that, based on our judgment, were vaguely-worded or secondary (i.e., contingent on the accomplishment of other goals), and any project whose principal goal would not result in a direct air service improvement (e.g., grant was awarded to complete a study). We categorized the 75 projects as “Establish Service,” “Promote Service,” or “Expand Service,” as defined below in table 8, or as a multi-goal combination of these goals. We subsequently excluded 9 grants from our

Page 48: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 43 GAO-19-172 Small Community Air Service Development

scope based on factors or circumstances unique to those airports or their projects.4

Table 8: GAO’s Categories and Definitions for Small Community Air Service Development Program Grants

Grant goal Definition Establish service To attract air service to a new destination or from an airline type

(i.e. low cost or ultra-low cost) not currently serving the airport. Promote service To promote existing or scheduled-to-start service. Expand service To attract larger planes, additional flights to a destination already

served, or expand from seasonal to year-round service.

Source: GAO Analysis. | GAO-19-172

For this effort, one analyst reviewed project documentation to identify and categorize each project’s goal(s), and another analyst verified the categorization. Where differences existed, the analysts discussed them and reached agreement.

Next, we collected data to evaluate project success within each grant category using airline schedule and DOT flight data from calendar year 2009 through 2019 including scheduled departures, passenger enplanements, and aircraft equipment. We obtained these data from Diio, a private contractor that provides online access to U.S. airline financial,

4To ensure a fair and consistent assessment of each grant project, we initially identified a population of 75 fiscal year 2010–2014 grants awarded by DOT that had been closed or had expired as of December 31, 2017. We subsequently excluded 9 grants from this analysis. We excluded 5 grants because their projects were studies or were related to implementing new airport facilities: Fort Wayne International Airport (fiscal year 2010), Kearney Regional Airport (fiscal year 2010), Keene Dillant-Hopkins Airport (fiscal year 2010), Fort Collins-Loveland Municipal Airport (fiscal year 2011), and Sanford Seacoast Regional Airport (fiscal year 2013). We excluded an additional 4 grants because of individual airport circumstances. Specifically, we excluded Sioux Gateway Airport (fiscal year 2010) because it became an Essential Air Service airport during its grant period, making it ineligible for the revenue guarantee for which its grant had been awarded. We removed Arnold Palmer Regional Airport (fiscal year 2011) from our scope because its air service began too late in the grant period to compare to 12 months of service before the grant. We removed Newport News/Williamsburg International Airport (fiscal year 2012) because at the time of our review the airport was under investigation by the Department of Transportation Inspector General’s Office, the Internal Revenue Service, and the Virginia State Police. Finally, we removed the Pelican Consortium (fiscal year 2013—Angoon Airport, Pelican Airport, Tenakee Springs Sea Plane Base, Kake Airport, and Elfin Cove Airport) based on our determination that Alaskan airports do not have comparable airports in the continental United States, given the unique circumstances of their air travel.

Page 49: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 44 GAO-19-172 Small Community Air Service Development

operational, and passenger data. To assess the reliability of these data, we reviewed the quality control procedures used by Diio, interviewed DOT officials responsible for data collection efforts, and subsequently determined that the data were sufficiently reliable for our purposes.

We determined an appropriate time period for analysis and established individual metrics of success for each project goal. Following the definitions defined below, one analyst made the initial determination of success and a second analyst concurred.

To evaluate the extent to which “Establish Service” projects were successful in attracting new service, we analyzed schedule data on scheduled departures. For example, if a grant project aimed to add service to a city that the airport did not presently serve, we analyzed schedule data to see if flights to that city began during the grant period. We first assessed success using an analysis period that began the month the airport signed the grant agreement and ended the month the grant was closed or had expired. For projects we deemed successful, we also assessed the sustainability of their success at 12 months and 24 months after the grant ended. We referred to grant agreements, amendments, or closeout letters from DOT to determine each grant’s end date. We determined the extent of each project’s success in establishing service using the metrics defined in table 9 below.

Table 9: GAO’s Metrics of Success for “Establish Service” Grants

Rating Description Unsuccessful Airport was not able to establish desired service within the grant

period. Partially Successful Airport was able to establish desired service within the grant

period, but did not sustain service through the last month of the grant period.

Successful Airport was able to establish desired service within the grant period, and in each year until the last month of the grant period, unless the service is seasonal. Seasonal service must be present in the appropriate season closest to the end of the grant.

Sustained “Successful” airport sustains service for exactly 12 or 24 consecutive months after the grant ends. For projects that established seasonal service, service must exist at some point within these time periods.

Source: GAO analysis. | GAO-19-172

Establish Service

Page 50: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 45 GAO-19-172 Small Community Air Service Development

To evaluate the extent to which “Promote Service” projects were successful, we utilized DOT T-100 data on passenger enplanements, which would be expected to rise if a project was successful in attracting more traffic to the airport. We distinguished projects in which airports were promoting their service in general from those in which airports were promoting particular routes.

Promote Service Grantees with Comparison Airports

For airports using their grant to promote their air service in its entirety (whether multiple routes or an established route), we first assessed success using an analysis period that began 12 months prior to when the airport signed the grant agreement and ended the month the grant was closed or had expired. For projects we deemed successful, we also assessed the sustainability of their success at 12 months and 24 months after the grant ended. We referred to grant agreements, amendments, or closeout letters from DOT to determine each grant’s end date. For each of these grantees, we created a comparison group of similar airports. Each comparison group is comprised of airports (1) whose passenger enplanements fell within a range of plus or minus 10 percent of the grantee airport’s enplanements in the calendar year prior to the grant, and (2) that did not have active grants during the period of analysis.

We calculated the change in enplanements that occurred at each grantee airport beginning 12 months before the grant period through the last 12 months of the grant period and compared it to the change in enplanements in its group of comparison airports over the same time period. We determined the extent of each project’s success in promoting service using the metrics defined in table 10 below.

Promote Service

Page 51: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 46 GAO-19-172 Small Community Air Service Development

Table 10: GAO’s Metrics of Success for “Promote Service” Grants with Comparison Airports

Rating Description Unsuccessful The airport’s enplanements did not increase by at least 5

percent in the period of analysis, and the airport’s percent change was within 5 percentage points of the comparison group’s percent change in enplanements.

Partially Successful The airport’s enplanements increased by at least 5 percent in the period of analysis, but the comparison group’s percent change were at least 5 percentage points greater than the airport’s percent change in enplanements—or— The airport’s enplanements did not increase by at least 5 percent in the period of analysis, but the comparison group’s percent change was less than 5 percentage points greater than the airport’s percent change in enplanements.

Successful The airport’s enplanements increased by at least 5 percent in the period of analysis, and the percent change was 5 percentage points or more than the comparison group’s percent change.

Sustained “Successful” airport sustains enplanements at a level at least 5 percent higher than the first 12 months of service/grant and the percent change in enplanements is 5 percentage points or more than the comparison group’s enplanements for 12 or 24 consecutive months after the grant ended.

Source: GAO analysis. | GAO-19-172

Promote Service Grantees without Comparison Airports

For airports using grants to promote one of multiple routes, or their only route was new, we first assessed success using an analysis period that began the month the airport signed the grant agreement and ended the month the grant was closed or had expired. For projects we deemed successful, we also assessed the sustainability of their success at 12 months and 24 months after the grant ended. We referred to grant agreements, amendments, or closeout letters from DOT to determine each grant’s end date. We calculated the change in enplanements at the grantee airport beginning from the first 12 months of the grant (or the first 12 months of service after the grant started) to a maximum of 24 months after the grant closed. We determined the extent of each project’s success in promoting service using the metrics defined in table 11 below.

Page 52: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 47 GAO-19-172 Small Community Air Service Development

Table 11: GAO’s Metrics of Success for “Promote Service” Grants without Comparison Airports

Rating Description Unsuccessful Airport did not increase passenger enplanements by at least 5

percent in the period of analysis. Partially Successful Not applicable because comparison groups were not used. Successful Airport increased passenger enplanements by 5 percent or

more in the period of analysis. Sustained “Successful” airport also sustained passenger enplanements at

a level at least 5 percentage points higher than first 12 months of service/grant for 12 or 24 consecutive months after the grant ended.

Source: GAO analysis. | GAO-19-172

To evaluate the extent to which “Expand Service” projects were successful, we analyzed schedule data on flight frequency and aircraft equipment to determine whether the grantee airport was able to expand its service. For example, if a grant project aimed to expand seasonal service to daily, year-round service, we analyzed schedule data to determine if service expanded during the grant period. We first assessed success using an analysis period that began the month the airport signed the grant agreement and ended the month the grant was closed or had expired. Because we determined that no “Expand Service” projects were successful, we did not assess whether their results were sustained after the grant ended. We referred to grant agreements, amendments, or closeout letters from DOT to determine each grant’s end date. We determined the extent of each project’s success in establishing service using the metrics defined in table 12 below.

Expand Service

Page 53: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 48 GAO-19-172 Small Community Air Service Development

Table 12: GAO’s Metrics of Success for “Expand Service” Grants

Rating Description Unsuccessful Airport did not achieve service expansion at any point in the grant

period. Partially Successful Airport achieved service expansion within the active grant period

but could not sustain expanded service through the last month of the grant.

Successful Airport achieved service expansion within the active grant period and sustained expanded service through the last month of the grant.

Sustained “Successful” airport also sustained service expansion 12 or 24 months after grant ended.

Source: GAO analysis. | GAO-19-172

Our scope includes two fiscal year 2010–2014 grants awarded to airport consortia, in which each airport had a “Promote Service” goal. We evaluated each airport in the consortium individually. Additionally, six grants were a multi-goal combination of “Establish Service,” “Promote Service,” or “Expand Service” projects. In these cases, we first evaluated each project goal or each airport within a consortium individually, and then evaluated the success of the overall project. Successful projects accomplished all of their goals, partially successful projects were at least partially successful in accomplishing one goal, and unsuccessful projects did not accomplish any of their goals.

Consortia Grantees and Grant Projects with Multiple Goals

Page 54: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 49 GAO-19-172 Small Community Air Service Development

We evaluated whether 60 fiscal year 2010–2014 grantees experienced an increase in connectivity during the grant period.5 Connectivity is a measure of an airport’s degree of access to the global air transportation system. We used a modified version of an International Air Transport Association (IATA) measure, in which an airport’s connectivity is calculated as the sum of the number of available seats to each destination from the airport, weighted by the size—based on annual enplanements—of the destination airport.6

We compared the change in connectivity at each grantee airport—the difference between its pre-grant connectivity and its connectivity at the end of the grant period—to the corresponding changes at a comparison group of at least 3 similarly sized airports. Each comparison group is comprised of airports that did not hold active grants during the period of analysis, and whose enplanements were within a range of plus or minus 10 percent of the grantee airport’s annual enplanements in the calendar year prior to the grant period. The period of analysis begins in January of 5For our analysis of airport connectivity changes, we excluded 15 grants from the population of 75 fiscal year 2010–2014 grants awarded by DOT that had been closed or had expired as of December 31, 2017. We excluded 3 grants because they were awarded to consortia of airports: Mississippi Consortium (fiscal year 2013—Meridian Regional Airport, Mid-Delta Regional Airport, and Hattiesburg-Laurel Regional Airport), Missouri Department of Transportation Consortium (fiscal year 2010—Columbia Regional Airport, Joplin Regional Airport, Kirksville Regional Airport, and Waynesville Regional Airport), and Pelican Consortium (fiscal year 2013—Angoon Airport, Pelican Airport, Tenakee Springs Sea Plane Base, Kake Airport, and Elfin Cove Airport). Next, we excluded 6 grants because their community airports did not have scheduled commercial air service in the year prior to the grant, which made creating comparison groups infeasible: Pike County Airport-Hatcher Field (fiscal year 2011) Auburn/Lewiston Municipal Airport (fiscal year 2011), Los Alamos Airport (fiscal year 2012), Arnold Palmer Regional Airport (fiscal year 2011), Keene Dillant-Hopkins Airport (fiscal year 2010), and Sanford Seacoast Regional Airport (fiscal year 2013). Additionally, we excluded 4 grants that had 2 or fewer airports in their comparison group of airports: Central Wisconsin Airport (fiscal year 2010), Evansville Regional Airport (fiscal year 2011), Topeka Regional Airport (fiscal year 2012), and Ogden-Hinckley Regional Airport (fiscal year 2012). We excluded Sioux Gateway Airport (fiscal year 2010) because it became an Essential Air Service airport during its grant period, making it ineligible for the revenue guarantee for which its grant had been awarded. Finally, we excluded Newport News/Williamsburg International Airport (fiscal year 2012) because at the time of our review the airport was under investigation by the Department of Transportation Inspector General’s Office, the Internal Revenue Service, and the Virginia State Police. 6IATA’s analysis uses a connectivity measure based on the number of available seats to each destination served for the first week in July in each year between 1996 and 2005. We modified IATA’s measure to include available seats per calendar year rather than per week in our period of analysis.

Connectivity

Page 55: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 50 GAO-19-172 Small Community Air Service Development

the calendar year before the year in which the grant agreement was signed and ends with the calendar year the grant closed or expired.

To develop this measure of connectivity, we used DOT T-100 data on passenger enplanements and available seats for each grantee and comparison airport. We calculated a weight for each airport by dividing its total annual enplanements by the annual enplanements at Hartsfield-Jackson Atlanta International Airport, the largest airport in the world by this measure in each year of our analysis. That is, all airports were scaled for their size in comparison to the largest U.S. airport. For each grantee and comparison airport, we then multiplied the available seats to each destination served by the destination airport’s weight for the appropriate calendar year. Finally, for each grantee and comparison airport, we summed these weighted seats to create a connectivity measure for the year prior to the grant and the year the grant ended.

We then calculated each airport’s percent change in connectivity by subtracting its pre-grant connectivity value from its grant-end connectivity value, then dividing the difference by the pre-grant connectivity value and multiplying by 100. For each comparison group of airports, we calculated an average connectivity measure for the year prior to the grant and the last year of the grant, and then calculated the percent change in connectivity in the same manner as described for grantee airports. If a grantee airport’s change in connectivity was at least 5 percentage points greater than the equivalent change in the average measures for its comparison group, we determined that it experienced an increase in connectivity.

To identify factors that affect the success of grant projects, we interviewed officials from the 20 selected fiscal year 2010–2014 grantee airports identified above, as well as officials from four grantee airports where we also conducted site visits, and a judgmental sample of 13 stakeholders in small community air service—including three academic experts and researchers, and representatives from two industry associations, four aviation consulting firms, and four airlines.7 From the information provided in these interviews, we identified eight factors that affected project success, including: community demand for air service, 7The stratified, random sample of grantee airports includes 10 airports whose projects we determined were successful, 5 airports whose projects we determined were partially successful, and 5 airports whose projects we determined were unsuccessful.

Factors Affecting Grant Project Success

Page 56: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 51 GAO-19-172 Small Community Air Service Development

distance to other airports, a strong or growing local economy, successful existing service at an airport, community support for the grant project, quality airline service, an airline letter of support for the grant project, and low airport operating costs.

We were able to identify proxy measures for six of the eight elements identified from interviews.

• Demand for air service: We used data on the population within a 60-minute drive of the grantee airport, obtained from the DIIO-MI database catchment mapper, as a measure of community demand for air service. The data were aggregated from the zip code level for each community.

• Distance to other airports: We used data on the distance from each grantee airport to the nearest medium or large hub airports.

• Strong/growing local economy: We used data on both income per capita and establishments within a 60-minute drive of the grantee airport to represent the strength of the local economy. These data elements were gathered from the DIIO-MI database catchment mapper. The data were aggregated from the zip code level for each community. These data points do not capture a “growing” economy, as the data are only available for one year.

• Current successful service: We used an airport’s status as a small hub to represent its existing amount of service. For “Promote Service” projects, we also used an airport’s EAS status to represent the amount of current service at an airport. In fiscal years 2010–2014, EAS airports were only eligible for grants to market or promote their EAS service.

• Community support for the project: We used a community’s cash contribution to a grant project as a measure of its support. We obtained this information from fiscal year 2010–2014 grant agreements signed with DOT. While many communities also include in-kind contributions, they are not all quantified in the grant agreements, and we therefore did not include these data.

• Airline letter of support: We used a binary variable, where the value “1” indicates a community had an airline letter of support. We gathered this information from grant applications, DOT’s documented evaluation of fiscal year 2010–2014 grants, and the public docket on regulations.gov.

Page 57: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix I: Objectives, Scope, and Methodology

Page 52 GAO-19-172 Small Community Air Service Development

Using these measures, we determined whether any of the factors affecting project success identified through interviews correlated with the outcomes (unsuccessful, partially successful or successful) of the “Establish Service” and “Promote Service” projects we evaluated. For the purposes of this analysis, we assigned a “1” value to successful and partially successful projects and a “0” value to unsuccessful projects. We evaluated each goal of a multi-goal project individually—assigning a “1” or “0” value depending on our earlier evaluation of its success. For a project, for example, with “establish service” and “promote service” goals—both evaluated as successful—we assigned a “1” value for that project in each analysis. We removed consortium projects from this analysis, since the demographic data are dependent on one airport, and consortium projects include multiple airports. While the characteristics associated with project success may be best understood using an analysis that considers multiple factors at once, due to the limited number of cases such an analysis was not possible. Instead, we used a correlation analysis that measures the strength and direction of the linear relationship between two variables without controlling for the effects of other characteristics. Coefficient variables take a value between negative 1 and 1. The magnitude of the correlation coefficient determines the strength of the correlation. A correlation coefficient of zero would indicate that there was no relationship between the variables. A correlation coefficient close to 1 would indicate a strong positive relationship, while a correlation coefficient close to negative 1 would indicate a strong negative relationship.

We conducted this performance audit from July 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Page 58: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix II: Federal Agencies, Airports, and Stakeholders in Small Community Air Service GAO Interviewed

Page 53 GAO-19-172 Small Community Air Service Development

Table 13: Federal Agencies, Airports, and Stakeholders in Small Community Air Service GAO Interviewed

Federal Agencies, Airports, and Stakeholders in Small Community Air Service U.S. federal agencies Department of Transportation Airports Auburn-Lewiston Airport (Auburn, ME) Bangor International Airport (Bangor, ME) Billings Logan International Airport (Billings, MT) Bozeman Yellowstone International Airport (Bozeman, MT) Branson Airport (Hollister, MO) Burlington International Airport (Burlington, VT) Casper/Natrona County International Airport (Casper, WY) Cherry Capital Airport (Traverse City, MI) Corpus Christi International Airport (Corpus Christi, TX) Delta County Airport (Escanaba, MI) Dothan Regional Airport (Dothan, AL) Fort Wayne International Airport (Fort Wayne, IN) Hector International Airport (Fargo, ND) Huntsville International Airport/Port of Huntsville (Huntsville, AL) Idaho Falls Regional Airport (Idaho Falls, ID) Inyokern Airport (Inyokern, CA) Ithaca Tompkins Regional Airport (Ithaca, NY) Lemhi County Airport (Salmon, ID) Manhattan Regional Airport (Manhattan, KS) Mid-Ohio Valley Regional Airport (Parkersburg, WV) Ogden-Hinckley Airport (Ogden, UT) Pitt-Greenville Airport (Greenville, NC) Pocatello Regional Airport (Pocatello, ID) Punta Gorda Airport (Punta Gorda, FL) Redmond Municipal Airport (Redmond, OR) Richmond International Airport (Richmond, VA) Santa Barbara Municipal Airport (Santa Barbara, CA) South Bend Airport (South Bend, IN) St. Cloud Regional Airport (St. Cloud, MN) Topeka Regional Airport (Topeka, KS) Tupelo Regional Airport (Tupelo, MS) Walla Walla Regional Airport (Walla Walla, WA)

Appendix II: Federal Agencies, Airports, and Stakeholders in Small Community Air Service GAO Interviewed

Page 59: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix II: Federal Agencies, Airports, and Stakeholders in Small Community Air Service GAO Interviewed

Page 54 GAO-19-172 Small Community Air Service Development

Federal Agencies, Airports, and Stakeholders in Small Community Air Service

Airports Waterloo Regional Airport (Waterloo, IA) Watertown International Airport (Watertown, NY) Yakima Air Terminal – McCallister Field Airport (Yakima, WA) Yuma International Airport (Yuma, AZ) Industry associations Airport Council International – North America American Association of Airport Executives Academic experts and researchers Russell W. Mills, Ph. D., Bowling Green State University Michael D. Wittman, Massachusetts Institute of Technology International Center for Air Transportation William S. Swelbar, Massachusetts Institute of Technology International Center for Air Transportation Aviation consultants Boyd Group International Sixel Consulting Group Trillion Aviation Volaire Aviation Consulting Airlines American Airlines Cape Air JetBlue SkyWest Airlines

Source: GAO. | GAO-19-172

Page 60: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix III: Comments from the Department of Transportation

Page 55 GAO-19-172 Small Community Air Service Development

Appendix III: Comments from the Department of Transportation

Page 61: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix III: Comments from the Department of Transportation

Page 56 GAO-19-172 Small Community Air Service Development

Page 62: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

Appendix IV: GAO Contact and Staff Acknowledgments

Page 57 GAO-19-172 Small Community Air Service Development

Heather Krause, (202) 512–2834 or [email protected]

In addition to the contact named above, the following individuals made important contributions to this report: Gerald L. Dillingham, Ph.D. (Director); Vashun Cole (Assistant Director); Justin Reed (Analyst-in-Charge); Amy Abramowitz; David Hooper; Delwen Jones; Kelsey Kreider; Alex Lawrence, Jr.; Bonnie Pignatiello Leer; SaraAnn Moessbauer; and Joshua Ormond.

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

(102158)

Page 63: GAO-19-172, SMALL COMMUNITY AIR SERVICE ...Since fiscal year 2002, DOT has awarded 401 SCASDP grants totaling approximately $188 million to improve air service to small airports. GAO

The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website (https://www.gao.gov). Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to https://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, https://www.gao.gov/ordering.htm.

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov.

Contact FraudNet:

Website: https://www.gao.gov/fraudnet/fraudnet.htm

Automated answering system: (800) 424-5454 or (202) 512-7700

Orice Williams Brown, Managing Director, [email protected], (202) 512-4400, U.S. Government Accountability Office, 441 G Street NW, Room 7125, Washington, DC 20548

Chuck Young, Managing Director, [email protected], (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548

James-Christian Blockwood, Managing Director, [email protected], (202) 512-4707 U.S. Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548

GAO’s Mission

Obtaining Copies of GAO Reports and Testimony Order by Phone

Connect with GAO

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs

Strategic Planning and External Liaison

Please Print on Recycled Paper.