state of application security

7/31/2019 State of Application Security

1/28


2/28Page 1

Executive Summary ...................................... ........................................ ......................................... ....................................... ...................................... . 2Introduction And Survey Methodology............................................................................................................................................................. 3For Many, Application Security Is Not Yet A Mature Practice ............................................................................................................... 5 From Design To Production, Software Security Practices Need To Improve ................................................................................ 8There Is A Silver Lining Looming ...................................................................................................................................................................... 13Key Recommendations ........................................................................................................................................................................................... 20Appendix A: Methodology ..................................................................................................................................................................................... 21Appendix B: Demographics .................................................................................................................................................................................. 21Appendix C: Endnotes ............................................................................................................................................................................................. 26

2011, Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Information is based on best available

resources. Opinions reflect judgment at the time and are subject to change. Forrester , Technographics, Forrester Wave, RoleView,

TechRadar, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective

companies. For additional information, go to www.forrester.com. [1-HMGX0Z]
http://www.forrester.com/consulting


3/28Page 2

In November 2010, Microsoft commissioned Forrester Consulting to conduct a survey study of 150 North

American software development influencers. The purpose of the study is to understand the current state of

application security development practices and identify key trends and market directions for application

security.

From the late 1920s to his final arrest in 1952, Willie Sutton robbed more

than 100 banks and stole more than $2 million. When a reporter asked him

why he robbed banks, he responded, "Because that's where the money is."

Today, the money is in software applications thats where companies

process their most sensitive data from credit card numbers to customer and

employee information as well as trade secrets. Identity, financial, and

intellectual property theft fuel an underground cyber economy estimated to

be in the billions of dollars.

Our survey found that while a majority of organizations have implemented some form of application security

measures, very few have put in place an end-to-end strategic approach that incorporates security throughout

the software development life cycle. In fact, the data suggests that organizations typically choose to transfer risk

from development to operations, where the remediation cost for vulnerabilities are the highest.

Our respondents also told us that the top driver for application security adoption is currently compliance.

Beyond that, the lack of time and lack of management support present major challenges for application security.

We also found that when it comes to application security, most organizations employ tactical measures and

point technologies; few subscribe to a holistic, prescriptive application security methodology. Similarly, not

many organizations employ metrics to track the success of application security initiatives. In fact, 72% of

everyone who answered the survey told us that their developers are not measured with security-related

metrics, and 57% do not send their security requirements downstream to guide quality and security testing.

Because of these reasons, the road to better application security is not an easy one along the way, loss of

efficiency is common as is the lack of measurable, positive return on investment (ROI).

Looking forward, as companies grapple with a more menacing and capable threat landscape, a growing set of

regulations and third-party requirements, and an unprecedented level of IT upheaval, they will have no choice

but to improve their application security postures. If organizations do not integrate security and privacy into the

development practices from the earliest stages, addressing it later will not only be more expensive but also

could be ineffective altogether. In this case, companies may find more things than just their applications at risk.

Forresters study yielded these key findings:

The study revealed that a significant percentage of

respondents do not employ a coordinated end-to-end approach to application security throughout their

development life cycle. Our data shows that 47% do not perform acceptance tests for third-party code, and


4/28


5/28Page 4

manufacturing, business services, and the public sector (see Figure 1). Those in the high-tech industry include

platform vendors (55%), independent software vendors (13%), original equipment manufacturers (11%),

original design manufacturers (9%), and value-added resellers (4%).

We drew our survey respondents from companies in many verticals with at least $500 million in annual

revenues. All respondents are either technologists or managers who are directly involved with their companys

application development processes. Of all respondents, sixty-three percent are software developers, 10% are

development managers, and the rest are distributed across architects, quality-assurance (QA) personnel,

security testing, project managers, product managers, and company executives (see Figure 2). Readers who are

interested in a more detailed description of respondent profiles should refer to Appendix A.

We conducted the online survey between November and December 2010, with 26 questions spanning software

development processes, security mechanisms, metrics, and organizational structures.

2%

2%

3%

4%

7%

8%

10%

64%

Government, education institution,

public sector

Business services and construction

Manufacturing

Other

Healthcare and biotech

Utilities and telecommunications

Financial services and insurance

High tech

Which industry vertical are you from?"


6/28Page 5

The first goal of the study is to understand how mature application security is as a general practice. We define

maturity as follows: A security practice is maturewhen it is well defined and able to respond proactively toemerging threats, with established technologies, well-known best practices, and established metrics to measure

and track performance.

We asked the respondents which application security mechanisms they have deployed within their

development life cycle. The answers suggest that many organizations do not practice security consistently

across the life cycle (see Figure 3). As an example, 47% reported that they do not perform security acceptance

tests for third-party code, 30% do not conduct static analysis on their code, 27% do not practice threat modeling

and usage scenario reviews during application design, and 19% do not gather security requirements based on

policies or risk assessment.

Which of the following most closely reflects your job function in relation to IT/software development?"

Sof tware development,

63%Development manager,10%

Sof tware architect, 9%

Sof tware qualityassurance (including

testing), 5%

Project managementresponsible for sof tware

development, quality, andsecurity, 5%

Company executive o rbusiness decision-makers,

5%

Product managementresponsible for sof tware

product d evelopment, 1%

Security testing, 1%


7/28Page 6

When we asked our respondents what was the most effective argument to convince their management to invest

in software security, the top answer was to meet our compliance requirements 25% of our respondents

selected this answer (see Figure 4).

Regulations like those in the payment card industry (PCI), the SarbanesOxley (SOX) Act, and the Health

Insurance Portability and Accountability Act (HIPAA) have requirements that either specifically call for the use

of software security mechanisms or indirectly do so through the mandate for vulnerability management. Its not

surprising to us that compliance is a big driver. But regulations typically lag behind threats, and they therefore

rarely represent the best practices. In fact, we often see applications that are deemed to be compliant to various

regulations being vulnerable to attacks. As a general rule of thumb, whenever compliance is cited as the top (and

sometimes the sole) driver, it means that the industry has not taken on the treatment of the problem in a

proactive or strategic manner. In other words, the practices are not yet mature.

3%

5%

14%

17%

19%

21%

27%

30%

47%

A prescriptive security incident response plan or o perational securityplan for production code

Developer (tester) training

Accountability and incentive structures to promote so ftware security

Quality or security gate in testing

Risk or policy -based security requirement definition

Archive release environments and activities as part of a secure release

process

Threat modeling and usage scenario review

Static analysis

Stringent security tests prior to acceptance of t hird-party code

Percent of respondents who said they do not currently employ the following measures for improving code security in

their organization


8/28Page 7

It is therefore not surprising when asked which prescriptive or process-based application security methodology

they use in development that our respondents returned answers that point to a decidedly tactical practice.

Forty-six percent said that they follow some form of homegrown security methodologies. The CMM and SDL

methodologies are ranked a distinct second and third (see Table 1). 3 Fourteen percent reported that they do not

follow any prescriptive methodologies.

25%

21%

13%

12%

12%

6%

6%

3%

1%

To meet our compliance requirements

It is cheaper to f ix bugs earlier in the development life cycle

Economic impact o f security breaches

A risk-centric argument

Customers require us to demonstrate secure development practices

We havent convinced our management to invest in sof tware

security measures yet

Its a competitive dif ferentiator in the market

Dont know

We need g reater accountability f or third-party code and greatertransparency for our software supply chain

What is the most effective argument in convincing your management to invest in software security measures during

the development life cycle?


9/28Page 8

In addition, when asked whether they use any list-based approaches to achieve software security, nearly 50%

said that they use a list of approved development tools and option settings (e.g., compiler options). A secure

development guideline was also a popular choice, garnering 42% of the votes. Surprisingly, OWASP top 10 and

SANS top 25 did not receive many mentions. This, combined with the fact that 46% use homegrown application

security methodologies, suggests that an overwhelming majority of companies treat application security with

approaches of varying quality few are tackling the issues systematically and in a measured way.

In the study, we asked in-depth questions of individual application security mechanisms, spanning the entire

software life cycle, from requirement generation to production. Our intention is to study practitioner priorities

Which prescriptive or process-based software security methodology do you use?"

Software security % of respondents

We have developed ourown sof tware security

methodology46%

CMM or CMMI 20%

SDL 15%

Other 2%

OpenSAMM 1%

DISA STIG (foroperations)

1%

We do not use any suchmethodology

14%

Dont know 16%


10/28Page 9

as well as the rationale and motivations behind them. The questions provide rich analysis of an organizations

software security efforts, adoption strategies, challenges that they face, and efficacy of current practices.

A detailed study of the data reveals that even though some practices are more mature and more widely adopted

than others, overall application security remains an uneven field in software development houses. More

specifically:

We asked our respondents to rank the

effectiveness of various security practices in the requirement definition phase. The majority of them, 54%,

believe that having a comprehensive set of security and privacy policies to start the requirement definition

is an effective practice for improving code security. Beyond that, however, 50% do not believe in the

benefits of using a risk-based approach to drive security requirements. Similarly, 57% do not believe in

communicating security requirements downstream to guide testing and QA efforts (See Figure 5). The

data directly shows the gap between development and testing as well as the lack of risk-based approaches.

Earlier data shows that 27% of

organizations are not doing threat modeling and usage scenario reviews during design (see Figure 3).

When asked whether typical design-phase security measures were effective to promote code security,

For security and privacy requirements definition, please rate how effective the following items are for improving

security of the code and reducing downstream work that arises due to security vulnerabilities.

9%

18%

19%

27%

18%

25%

31%

27%

30%

24%

29%

22%

17%

17%

12%

13%

7%

3%

2%

1%

18%

13%

7%

9%

0% 25% 50% 75% 100%

Derive requirements exclusively from industry standards (e.g.,OWASP top 10, SANS top 25, BSIMM)

Send the security and privacy requirements downstream for securitytesting

Derive security and privacy requirements based on the app lication

risk profile, the likelihood of threats, and your risk tolerance level

Start the requirements definit ion phase with comprehensive securityand privacy policies

Most ef fective (5) 4 3 2 Least effective (1) We do not currently have this


11/28Page 10

52% said that they do not believe in the effectiveness of risk-based architecture views, and 59% do not

believe in threat modeling and usage scenario review. This is yet another data point that suggests the lack

of risk-based approaches (or at least the lack of those that are being used effectively) in security design.

In the

implementation phase, our respondents selected source code management, manual code reviews, and a

secure coding guideline as the most effective security measures. In the security community, however,

technologies like static analysis and, to some extent, fuzzing performed by developers are widely regarded

as far more effective in discovering application vulnerabilities. But our survey data shows that these

technologies are not widely deployed (see Figure 6). This result, perhaps, is in part due to the fact that

most application security activities are driven by compliance, which may emphasize secure coding

guidelines (as in PCI) and source code management. One can also infer from this data that organizations

are investing in easy-to-implement application security measures, such as secure coding guidelines and

manual reviews, but technologies like static analysis that may require more extensive changes to existing

development processes are not being adopted.

For security measures taken during implementation, please rate how effective the following items are for improving

security of the code and reducing downstream work that arises due to security vulnerabilities.

4%

8%

9%

9%

11%

11%

19%

20%

20%

10%

25%

22%

28%

28%

24%

29%

32%

32%

18%

24%

36%

25%

29%

30%

29%

30%

29%

20%

10%

18%

16%

14%

13%

13%

14%

12%

7%

3%

2%

3%

6%

1%

1%

3%

3%

41%

30%

12%

18%

12%

21%

9%

1%

3%

0% 25% 50% 75% 100%

Binary code analysis

Static analysis technologies

Developers performing fuzzing tests on the code

Developers using penetration testing too ls (e.g., black-box scanning)

Manual penetration testing

A library of approved or banned functions

A secure coding guideline

Manual code reviews

Systematic source code management and tracking

Most effective (5) 4 3 2 Least effective (1) We do not currently employ this type of measure


12/28Page 11

For those respondents who indicated

that they remediate at least certain security flaws prior to release, we asked what process they follow to

triage and prioritize vulnerability remediation. Sixty percent of all respondents said that they use a risk-

based approach in vulnerability triage and for the prioritization of remediation. This is encouraging,

especially considering how little risk-based approaches are used in other parts of the development life

cycle.

When asked about

their code release practices and relevant security measures, 52% find the use of a formal sign-off step (to

indicate that all secure measures have been met) to be most effective, and 49% find the most value in a

final penetration test prior to release. Only 40% saw the merit of archiving pertinent information and data

for the sake of future incident and bug responses. Typically, one would archive specifications, source code,

binary builds, documentation, incident response plans, and licensing and service terms (if third-party

software). Archiving the information allows a baseline for effective incident triage and response. Again,

this result points to the lack of mature security considerations in the respondents software release plan.

For servicing software in production, we asked the

respondents to rate the effectiveness of various security measures. The majority of them reported that

they saw the benefits of an established bug-reporting channel, application monitoring/diagnostics

mechanisms, and a regular, well-communicated software patch program. The fact that more organizations

are practicing secure operations but substantially fewer of them are incorporating security consistently

into other parts of the development life cycle suggests that organizations choose to transfer risk from

development to operations, where remediation costs are the highest.4

We also looked at whether organizations institute special accountability and incentive structures to promotesoftware security initiatives and whether these measures indeed help organizations in achieving their goals. We

found that:

In our interaction with software

development and software security professionals, Forrester sees much inefficiency in the way

development and test teams typically interact. Often there is much time wasted on going back and forth

between development and testing on whether a test finding is a true software bug. This inefficiency

underlines many of the challenges for software security you need to change the existing development

processes if you want to produce efficient and effective implementation of security measures. We asked

our respondents whether their companies use any special incentive programs to encourage development

and testing personnel to work together in a more efficient fashion. Sixty-one percent said that they had nosuch program. Only 43% reported some form of organizational process to strengthen the working ties

between development and testing (see Figure 7).


13/28Page 12

When asked whether their developers are measured

with security-related metrics, 72% of all respondents and, more specifically, 76% of all developers who

answered the survey said no. This is a testament that, at a larger scale, developers are often not given the

proper incentives to respect security initiatives and goals.

We asked our respondents whether they use any

concrete metrics to calculate ROI for incorporating security measures within software development.

Nineteen percent said that they do not use any such metrics. Twenty-six percent reported the use of only

software time to market as the ROI metric.

Our study uncovered an interesting data point that highlights the different perspectives of (hence attitude

thereof) the different stakeholders regarding application security. When asked what the top challenges are in

implementing a secure development program, developers, development managers, project managers, and

testers all cited lack of time to perform security tasks as a top challenge . However, among the seven executives

What incentive mechanism do you use to ensure that your development team and testing teams work closely

together?"

61%

23%

11%

9%

5%

1%

We have no special incentive mechanism to encouragedevelopment and test work together

We pair developers and testers and measure the combined unit witha set of quality metrics, including security metrics. This ensures that

they work together to achieve overall software quality object ives

We have testers rate how testable a developers code is and rewardthe developer accordingly

We have developers rate the test results (e.g., how many f alsepos itives, how easy it is to digest the test outcome) and reward

testers accordingly

Dont know

Other


14/28Page 13

who answered our survey, zero selected lack of time as a challenge. Rather, they cited lack of security

expertise and lack of funding as top challenges.

This data is most telling, as it suggests a potential chicken-and-egg problem: If executives do not realize that

lack of time is a real concern, they will continue to set goals for development that are non-commesurate with

security, such as emphasizing time to market above all things. As a direct consequence, this will continue to

encourage developers to overlook security for other performance metrics. As a result, executives view of

developers lacking security expertise might be perpetuated.

The data in our study, from various fronts, painted a picture of a software industry that does not yet use mature

security practices. Many of the top challenges facing developers and application security professionals, including

the lack of time and lack of funding, arise from failure to align application security objectives with business

goals.

In an effort to understand the business impact of employing application security mechanisms, we asked our

respondents what ROI metrics they keep and whether theyve seen positive ROI resulting from application

security deployment (see Figure 8). As shown, 26% of those who track the amount of developers time spent in

post-development bug chasing and remediation saw that the time decreased as a result of implementing

software security practices, while another 24% saw that the metric increased. Similarly, 40% (out of 42 that

kept track of testers time spent in regression testing) observed an increase in this metric, while 19% saw that it

decreased.


15/28Page 14

The overall ROI metrics do not show clear evidence that there is positive ROI across the board for incorporating

security within development. However, when we took a deeper look at some of the data points with correlated

analysis, we saw a few interesting and promising results.

We took a look at the set of the population using prescriptive

application security methodologies. It turns out that those practicing SDL specifically reported visibly

better ROI results than the overall population. Unlike point technologies, SDL advocates a coordinated

approach to application security throughout the life cycle, and its emphasis is on a set of processes that

supports such coordination. We can potentially extrapolate that a coordinated approach to application

security is what drives positive ROI. Table 2 shows the ROI with noticeable differences (the other metricsare similar).

Two practices that have seen

particular success in practice have to do with integrating development and testing processes. The first one

includes establishing a common quality and security standard between development and the testing team

that stipulates what constitutes a flaw or a security flaw . Such a common standard eliminates

What impact has the implementation of software security during development had on the following metrics?

24%

40%

23%

53%

38%

35%

31%

32%

25%

25%

26%

19%

30%

14%

9%

15%

10%

16%

8%

28%

0% 25% 50% 75% 100%

Amount of developers time in post-development bug-chasing and remediation (N = 68)

Amount of testers time spent on regression test ing (N= 42)

Amount of human time spent in incident response,patch release, and customer servicing (N = 57)

Time to market for the sof tware (N = 36)

Opportunity co st f or developer time (N = 2)

Increased No change Decreased Dont know


16/28


17/28Page 16

ROI metricThose who send security and privacy

requirements downstream to testingOverall

Amount of developers time in post -development bug-chasing and

remediation

37% reported decreased

(N=37)


(N=68)

Amount of human time spent in incident

response, patch release, and customerservicing


(N=30)


(N=57)

ROI metricThose who send security and privacy

requirements downstream to testingOverall

Amount of developers time in post -development bug-chasing and

remediation


(N=35)


(N=68)

Amount of human time spent in incidentresponse, patch release, and customer

servicing


(N=30)30% reported decreased

(N=57)


18/28Page 17

If we only look at the

ROI results reported by those who rated their security measures as very effective or effective, we also

see better ROI results than those reported by the overall population. Figure 9 and Figure 10 show the

comparative results. In both figures, the subset of populations who have implemented effective security

measures reported better RoI results than the overall survey population. In Figure 9, notice that a much

larger percentage of those who practice effective security maintenance reported seeing a decrease in

developers time spent on post-development bug-chasing work than reported by the overall population.

Similar results can be seen with human time spent on incident response.

ROI metricThose who send security and privacyrequirements downstream to testing

Overall

Amount of testers time spent on

regression testing 7 out of 24 reported dec reased 8 out of 42 reported dec reased


19/28Page 18

38%

34%

33%

33%

33%

32%

31%

26%

Those who practice eff ective secure maintenance of p roductio n code (N= 40)

Those who have an ef feci tve security and privacy requirementgathering process (N = 38)

Those who have effective accountability and incentive structures (N =33)

Those who p ractice effective secure release (N = 40)

Those who have eff ective secure design practices (N = 40)

Those who practice effective secure implementation (N = 47)

Those who practice eff ective security or quality testing (N = 35)

Overall (N = 68)

Types of respondents who reported a decrease in the amount of the developers time in post-development bug-

chasing and remediation


20/28Page 19

The RoI metrics results, as shown above, are one of the first concrete examples that show visible difference

between those who practice application security in a coordinated way and as an integral part of softwaredevelopment lifecycle versus those who do not.

Some of the data referenced above is not statistically significant. As such, we caution readers against making

general statements beyond this study. Nonetheless, they are interesting data points that are worth pondering

over.

Those who have eff ective accountability and incentive structures (N =21)

Those who have an ef fecit ve security and privacy requirementgathering process (N = 21)

Those who practice eff ective secure maintenance of p roductio n code (N= 27)

Those who have eff ective secure design practices (N = 27)

Those who practice eff ective secure release (N = 24)

Those who practice eff ective secure implementation (N = 31)

Those who practice effective security or quality testing (N = 22)

Overall (N = 42)

Types of respondents who reported a decrease in the amount of the testers time spent on regression testing

7

7

7

6

6

6

8

4


21/28Page 20


22/28Page 21

In this study, Forrester conducted an online survey of 150 North American software development influencers

and decision-makers in the US and Canada. Survey participants included leads in engineering, development,

product management, and product strategies. The study began in November 2010 and was completed in

December 2010.

US, 81%

Canada, 19%


23/28Page 22

2%

2%

3%

4%

7%

8%

10%

64%

Government, education institution,public sector

Business services and construction

Manufacturing

Other

Healthcare and biotech

Utilities and telecommunications

Financial services and insurance

High tech

Which industry vertical are you from?"


24/28Page 23

0%

16%

15%

11%

58%

$499 million USD or less

$500 million to $999 million USD

$1 billion to $2.49 billion USD

$2.5 billion to $5 billion USD

More than $5 billion USD

Which of the following most closely describes your companys total annual revenue?"


25/28Page 24

Does your firm conduct software development to supply either company business needs or customer needs (e.g.,

selling software products)?

91%

36%

23%

0%

0%

0%

Yes, we develop sof tware products or services

Yes, we develop the sof tware components fo r nonsof twareproducts

Yes, we are a sof tware outsourcer or a sof tware platformprovider (our customers may be development shops

themselves)

We rarely do sof tware development in-house

No, we dont do any sof tware development

Dont know


26/28Page 25

How are you involved in the software development efforts in your company (or within your department/organization)?"

76%

11%

8%

3%

2%

Im directly involved with software development, qualityassurance, and/or sof tware security; part o f a team

delivering systems and products

I manage a software development team (or teams)delivering systems and products

Im indirectly, but significantly involved, with sof twaredevelopment (e.g., reviewing deliverables and inf luencing

sof tware development teams and decisions

I am senior management or a decision-maker for anorganization that does so ftware development

My role is compliance, and I help to set our so ftwaredevelopment standards and practices based on comp liance

requirements


27/28Page 26

1 Microsoft pioneered the security development life cycle work. The Microsoft SDL documentation and process

description can be found on Microsofts website. Source: Microsoft (http://www.microsoft.com/security/sdl/).

2 CMM is a process improvement methodology that is broadly applicable to a diverse set of problems. It includes

security elements but is not designed to be a secure development methodology. The description of CMM and

CMMI can be found at Carnegie Mellon Universitys website. Source: Carnegie Mellon University

(http://www.sei.cmu.edu/cmmi/).

3 It should be noted that the CMMI methodology (CMMI now super cedes CMM) is designed to assess the

maturity of a development organization from a broad set of aspects. The methodology does include certain

elements for software security, but is not designed specifically to be a secure development methodology. As

such, it does not include a comprehensive set of secure development practices.

How would you characterize your firms current practices for software security ?"

57%

23%

10%

9%

1%

We have established extensive softwaresecurity technologies, processes, and metricsto ensure the production of secure sof tware

We care about software security primarilybecause of compliance reasons; our security

measures are driven by compliance

We have allocated signif icant investment forsof tware security too ls and technologies but

havent built out all of the relevant

organizational processes and metrics

Our sof tware security methodologies andstrategies for so ftware security are stillfo rming, and our practices at present are on

an as-needed basis

We care about software security in theory butdont currently use any software security

measures or allocate any investment for them


28/28

4 A NIST study found that fixing bugs earlier in the development life cycle is significantly cheaper than waiting

until later stages in the life cycle. Source: Gregory Tassey, Ph.D. The Economic Impacts of InadequateInfrastructure for Software Testing, May 2002 (http://www.nist.gov/director/planning/upload/report02-

3.pdf).

state of application security

Documents