brian zuckerman presented to

Brian Zuckerman

Presented to

COSEPUP Evaluation of the National Science

Foundation's Experimental Program to Stimulate

Competitive Research (EPSCoR) and Similar Programs

in Other Federal AgenciesDecember 17th, 2012

Results of the Assessment of theDefense Experimental Program to Stimulate

Competitive Research (DEPSCoR)

Study Origin and Timeline

• FY2008 Defense Authorization Act (Public Law 110-181), Section 241, instructed the Secretary of Defense to utilize a defense Federally Funded Research and Development Center (FFRDC) to carry out an assessment of the DEPSCoR program.

• Institute for Defense Analyses was tasked to conduct study February 2008

• Results briefed to Senate and House Armed Services Committee staff (SASC/HASC) November 2008

• Study results cleared for public distribution January 2009

Legislative Mandate for Study

1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expanded national research infrastructure

2. Activities consistent with statute

3. Assessment of program elements

4. Assessment of activities of state committees

5. Advantages and disadvantages of institution & state based formulas

6. Mechanisms for improving the management and implementation of the program

Descriptive Statistics

• Between 1993 and 2008, 729 total DEPSCoR awards– 546 individual Principal Investigators (PIs)

» 121 PIs with multiple awards» 42 with three or more» 1 PI with eight awards

– 22% of PIs have won 42% of awards

• 1993-2008 funding of $243 million– Decline after 2000 peak partially reversed in 2008 competition

• 27 states and territories (states) have been eligible for at least one year since program authorized in current form in 1995, plus Missouri (eligible in 1993)

– All eligible states except for the Virgin Islands have won awards

• 19% (5) of eligible states have won 35% of awards– Montana, Alabama, Oklahoma, Nebraska, South Carolina

• 7% (5) of institutions have won 28% of awards– Montana State, U. Nebraska-Lincoln, U. Wyoming, West Virginia U., U.

Arkansas

DEPSCoR Program Objectives

1. To enhance the capabilities of institutions of higher education in eligible states to develop, plan, and execute science and engineering [S&E] research that is competitive under the peer-review systems used for awarding federal research assistance

2. To increase the probability of long-term growth in the competitively awarded financial assistance that institutions of higher education in eligible states receive from the federal government for science and engineering research

The DEPSCoR State Share of DOD S&E Increased

Source: National Science Foundation (NSF) Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions and the DEPSCoR program database of funding totals by state and year

Note: Graph includes all states ever involved in DEPSCoR program 1995-2008

Note: The dotted lines represent linear regression models applied to the data

6%

8%

10%

12%

14%

16%

18%

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Year

Sh

are

of

Do

D U

niv

ersi

ty F

un

din

g t

o D

EP

SC

oR

Sta

tes,

A

dju

stin

g f

or

DE

PS

Co

R P

rog

ram

mat

ic F

un

din

g

R&D (Actual) R&D (Linear Model) Research (Actual) Research (Linear Model)

DEPSCoR Funding As a Percentage of DOD Funding in DEPSCoR States

• DEPSCoR has declined in importance as a source of funding for eligible states since 2000

Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions and the DEPSCoR program database of funding totals by state and year

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Year

Per

cen

tag

e o

f D

oD

Fu

nd

ing

in

DE

PS

Co

R-E

lig

ible

S

tate

s A

cco

un

ted

fo

r b

y D

EP

SC

oR

Pro

gra

m F

un

ds

Percentage of DoD Total University S&E R&D Percentage of DoD Total University S&E Research

“Success” Varied Among States

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

14.0%

16.0%

18.0%

20.0%

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Year

Sh

are

of

Do

D U

niv

ersi

ty S

&E

R&

D

Near or Above Threshold Rising Fast Middle Lagging

• DEPSCoR-eligible states fell into four groups:– 6 states Near or above threshold (AL, HI, LA, MS, NM, SC)– 9 states Rising fast (AK, ID, KY, ME, MT, NE, NV, ND, SD)– 6 states Middle (AR, DE, KS, OK, RI, TN)– 4 states + 2 territories Lagging (NH, PR, VT, VI, WV, WY)

Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, the IDA DEPSCoR database, and the DEPSCoR program database of funding totals by state and year

• DEPSCoR share of university S&E R&D funding varies substantially by state

• More than 60% for VT, WY

• “Rising fast” states’ (AK, ID, KY, ME, MT, NE, NV, ND, SD) DEPSCoR shares decline 2001-2005 compared with 1993-2000

0

AZ

0CA

0

CO

0

CT

0DC

0

FL

0

GA

0

GU

0IA

0IL

0

IN

0MA

0

MD

0

MI

0

MN

0

NC

0

NJ

0

NY

0OH

0

OR

0

PA

0

TX

0

UT0

VA

0VI

0

WA

0WI

1HI

2

RI

4

NM

5

DE

5TN

6

AK

6

MO

6

NH

12

SD

18

PR

26ME

28

LA

28ND

31

VT

33

ID

35

MS

36NV

37KY

38

AR

38

KS

38WV

41WY

47SC

49

NE

52OK

53AL

54MT

0 DEPSCoRs1-10 DEPSCoRs11-20 DEPSCoRs21-30 DEPSCoRs31+ DEPSCoRs

Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, the IDA DEPSCoR database, and the DEPSCoR program database of funding totals by state and year

Note: Average calculated as total DEPSCoR funds during eligible years divided by total DOD funds during eligible years

Average DEPSCoR Funding as a Fraction of DOD Funding by State

Expanded National Research Infrastructure (1)

• Involving new investigators:– Recent (2006-2008) cohorts had about 60% new PIs

– Most Army-funded DEPSCoR awardees (82%) had not previously received funding from Army Research Office (ARO)

– 56% of PIs had been funded by the NSF either previous to or within the same year of their first DEPSCoR award

• Training graduate students and postdoctoral fellows:– ARO and Office of Naval Research (ONR) data suggest that

awards fund about 1 PhD, 1 Master’s degree, 2 postdocs

• Building physical infrastructure: – DEPSCoR awards have supported purchase and maintenance

of equipment but data not collected systematically by services

Expanded National Research Infrastructure (2)

• Leveraging new funding for defense-related research is limited:

– 8% of non-DEPSCoR ARO awardees in DEPSCoR states received a DEPSCoR award before (or in the same year as) their first non-DEPSCoR ARO award

– 4 DEPSCoR awardees (less than 1%) won a DOD Multidisciplinary University Research Initiative (MURI) award after or in the same year as their first DEPSCoR award

» 2 of these investigators received non-DEPSCoR DOD funding before their first DEPSCoR award

• Leveraging other funding:– 63 DEPSCoR awardees (12%) received their first NSF funding

subsequent to their first DEPSCoR funding


1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expand national research infrastructure

2. Activities consistent with statute• Activities were found to be consistent with statute

3. Assessment of program elements• Available data on DEPSCoR program activities and outcomes are

insufficient for monitoring and evaluation purposes

4. Assessment of activities of state committees• State committees prioritized proposals that met state infrastructure

development goals and reflected the mission/research needs of DOD• Committee processes varied widely from state to state and limited and

variable data prevented detailed assessment



Approach Taken

• Assessment focused on comparing the current state-based formula with an institution-based criterion based on a maximum threshold for DOD S&E research funding

• As directed in the study legislative mandate, particular emphasis was given to supporting defense missions and expanding the nation's defense research infrastructure

– Since the legislative charge for the assessment does not specify how “expanding the nation’s defense research infrastructure” should be interpreted, the assessment considered advantages and disadvantages using a variety of possible interpretations

• Assessment also considered alternative criteria:– Indicators of state-level S&T capacity (SEI, Milken Institute)– Normalization by state population (Census)

• State-based and institution-based criteria can be combined– PI-level criteria are also possible

Effect of Institution Based Funding Threshold

• 77 institutions in 2008 DEPSCoR-eligible states (360 total) received nonzero research funding from DOD in 2005

– 38 Carnegie “Very High” or “High” research institutions in DEPSCoR states

• $5 million threshold would make 269 institutions eligible– Twelve of the 77 institutions in currently eligible jurisdictions (e.g., University of

Delaware, University of Nevada, Brown, Clemson, Vanderbilt, University of Nebraska) would become ineligible

• Considering only Carnegie “Very High” or “High” research universities, shift would increase number of eligible universities from 38 to 121

Maximum DOD Funding Threshold

All Universities With Non-Zero DOD Research

Funding in 2005

Eligible for the 2008 Competition

Potentially Eligible for the 2009 Competition

No limit 360 77 114

$10 million 316 75 109

$5 million 269 65 90

$3 million 231 55 76

$1 million 157 36 52

Definition of “Expanding National Research Infrastructure”

• If interpreted as increasing equity in funding among states or achieving state-level infrastructure goals, a state-based formula would be advantageous

– Current state-based formula for eligibility harnesses the state EPSCoR committees to coordinate infrastructure and capacity-building at the state level

– If intent is to increase the competitiveness of historically-underrepresented states, eligibility can easily be determined at a state level

• If interpreted as involving new investigators or institutions in defense-related research, an institution-based formula would be advantageous

– Allows targeting of programmatic resources toward investigators at institutions that have not historically built relationships with DOD

– Approach taken by late 1980s/early 1990s DOD Research Initiation Program

– While a state-based approach includes the flexibility to channel DEPSCoR proposals toward historically underrepresented universities or new investigators within an eligible state, the institution-based approach allows greater flexibility to target underrepresented universities and investigators throughout the entire country

• Could not be determined whether state-based or institution-based approach would elicit more qualified applications to support defense missions

– Larger number of eligible institutions implies more proposals, but quality indeterminate

Effect of State Population Normalization

• Green states were eligible in 2008, red states are “graduates”, blue states were never eligible

• Comparing top chart with bottom shows dramatic difference in order

• DEPSCoR graduates among highest per capita recipients

• Several DEPSCoR-eligible states (e.g., AK, DE, ND, RI, MT, SD) above average in funding per capita

$0

$50,000

$100,000

$150,000

$200,000

$250,000

$300,000

$350,000

State

FY

20

05

Do

D U

niv

ers

ity

S&

E R

&D

Fu

nd

ing

(K

$)

$0.00

$10.00

$20.00

$30.00

$40.00

$50.00

$60.00

$70.00

State

FY

20

05

Do

D U

niv

ers

ity

S&

E R

&D

Fu

nd

ing

Pe

r C

ap

ita

’05 DOD S&E R&D Funding, by State

’05 Per capita DOD S&E R&D Funding, by State

Other Capacity Indicators

• Some DEPSCoR-eligible states (e.g., NH, RI, DE, KS) above average in Milken Institute state S&T index

– Some “graduated” states below average

• Some DEPSCoR-eligible states (e.g., DE, KS, NE, NH, RI, VT) in the top or second quartiles for more than half of Science and Engineering Indicators’ seven R&D output measures

0

10

20

30

40

50

60

70

80

90

Milk

en In

dex,

200

8 Sc

ore

IDA Recommendations

• DOD should change the current process for review of proposals to focus more heavily on investigators’ future potential to conduct research rather than on their current research capabilities

• DOD program managers should be formally encouraged to serve as mentors and facilitators for DEPSCoR investigators seeking to engage in further defense-related research

• DOD should create data systems that will allow systematic tracking of DEPSCoR activities and outcomes

• Congress should re-examine and consider clarifying ambiguities in the DEPSCoR legislative mandate

• Once the DEPSCoR objectives have been clarified, redesign the program with a strategy for enhancing competitiveness at relevant level (e.g., individual, institution, state)

Some DEPSCOR Legislative Language Is Inconsistent

• Objectives specify that the research institution is the level at which competitiveness is to be enhanced but authorizing legislation also specifies that eligibility for DEPSCoR be determined at the state level

– 2008 change to eligibility criteria leaves ambiguous whether Congress intends DEPSCoR to target competitiveness at the institution or state level

• Objectives specify that goal is to increase probability of long-term growth in the competitively awarded financial assistance that institutions of higher education in eligible states receive from the federal government but eligibility determined based on DOD funding

– Statute does not discuss whether Congress intends DEPSCoR should focus narrowly on [6.1] research that is relevant to DOD missions and priorities although this is how program is run

Recommendations for Legislative Clarification

• To ensure that the program is implemented in a manner that meets current legislative priorities, Congress should clarify whether

– The program is intended to increase competitiveness for federal research funding in general or for particular types of research funding

– The program is intended to fund primarily basic research, primarily applied research, or a combination

– The primary unit at which competitiveness should be enhanced is the institution, state, or other (e.g., individual investigator)

• Eligibility criteria, funded activities, and other program elements should be structured in accordance with the program’s objectives

Competitiveness can be Enhanced at Different Levels

• DEPSCoR supports individual or small-group research projects and can therefore be understood to primarily target capacity-building at the level of the individual

– Though supporting training and purchase of equipment target capacity-building at level of the institution as well

• DEPSCoR also operates at state level (e.g., involvement of EPSCoR committees, state-based eligibility criteria)

• While it might be argued that institutional competitiveness depends on individual competitiveness and state competitiveness depends on institutional competitiveness, these dependencies are neither straightforward nor self-evident

Structure Program to Enhance at the Desired Organizational Level

• Assessment identified other programs that might serve as potential models for DEPSCoR

– State-level: NSF EPSCoR/Research Infrastructure Improvement (RII) Awards

» One per state; integrated into state S&T plan; EPSCoR committee involvement; capacity building rather than research

– Centers: NIH IDeA/Centers of Biomedical Research Excellence (COBRE)

» Large-team research, equipment, mentoring; milestones for transition to support by standard NIH mechanisms

– Investigator-level: DOD Research Initiation Program (RIP) or NIH/IDeA Academic Research Enhancement Award (AREA) Program

» Institution-based eligibility criteria; small research awards to individual investigators; build relationships/expertise to allow for transition to support by standard research mechanisms

Backup Slides

State Eligibility over Time

Jurisdiction 95 96 97 98 99 00 01 02 03 04 05 06 07 08

AL

AK

AR

DE

HI

ID

KS

KY

LA

ME

MS

MT

NE

NV

NH

NM

ND

OK

RI

SC

SD

TN

VT

WV

WY

PR

VI

Fiscal Year

Source: IDA analysis of DEPSCoR BAAs and DOD DEPSCoR press releases

Note: Red cells denote years in which jurisdictions were not eligible, and blue cells denote years in which states were eligible.

Note: Missouri, which was eligible only in the 1993-4 competition, was not included in the table.

No Obvious Correlation

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

14.0%

16.0%

18.0%

20.0%

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Year

Sh

are

of

Do

D U

niv

ersi

ty F

un

din

g t

o D

EP

SC

oR

Sta

tes,

Ad

just

ing

fo

r D

EP

SC

oR

Pro

gra

mm

atic

Fu

nd

ing

R&D (Actual) R&D (Linear Model) Research (Actual) Research (Linear Model)$0

$5

$10

$15

$20

$25

1993-4 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

Competition

Fu

nd

ing

(M

$)

0

10

20

30

40

50

60

70

80

90

Nu

mb

er o

f aw

ard

s

Funding (M$) Number of Awards

Number of DEPSCoR awards and program funding

Linear regression model of DEPSCoR state share of DOD university funding

DEPSCoR Emphasis is on Basic Research

• Research oriented towards developing operationally useful devices or components is not a legislatively-mandated program goal

• DEPSCoR projects mostly fund basic research– Not a statutory requirement– Administered by basic research organizations within Air Force

and Army, plus Office of Naval Research (ONR)» Interviews with DEPSCoR program officers suggest that more

applied research does not perform well in review– DEPSCoR program officers tend to track transitions to 6.2 or

6.3 research, but not further

• Two transitions to operational use were found

Two Transitions to Operational Use Were Found

• Ronald DeVore, University of South Carolina: Wavelet mathematics for image compression for tactical applications

– DeVore and colleagues collaborate with program managers at Naval Air Warfare Center NAWC at China Lake to deliver wavelet-based image processing platform

– Charles Creussere of NAWC implements wavelet-based image processing system for navigation in the Tomahawk Block II program

• Michael Pursley, Clemson University: wireless, mobile, distributed, multimedia communication networks

– Pursley and colleagues working since 1970s with ITT on tactical radio development

– Group used DEPScoR funding to support research that provided better anti-jam communications and greater multiple-access capability

– Research led to the Soldier Level Integrated Communications Environment (SLICE) wide band networking waveforms that have been integrated into the SINCGARS radio

brian zuckerman presented to

Documents