survey report a deep dive into data culture · respondents believe use data most effectively in...

© 2016 New Relic, Inc. | US +888-643-8776 | www.newrelic.com | www.twitter.com/newrelic | blog.newrelic.com

A Deep Dive into Data CultureSurprises and affirmations from the journey to a data-driven business

Survey Report

http://www.newrelic.com

https://twitter.com/newrelic

http://blog.newrelic.com


Table of ContentsINTRODUCTION 03

WHO WE SURVEYED 04

ON THE ROAD TO DATA-DRIVEN SOFTWARE SUCCESS 05

FULL SPEED AHEAD ON INNOVATION 11

JOB EXPECTATIONS NOW INCLUDE A DATA CULTURE 14

LOOKING FORWARD 16




Report

03

A Deep Dive into Data Culture: Surprises and affirmations from the journey to a data-driven business


IntroductionLet’s talk data. You love it, we love it. So indulge us for a moment,

please. IDC predicts that by 2020, there will be more than 44 zettabytes

(or 44 trillion gigabytes) of data in the ‘Digital Universe’.1 That’s a lot of

information. Already there’s enough that pretty much every company,

in nearly every industry, has the potential to access simply astounding

quantities of information. What remains to be seen—that is, what separates

the good from the great companies—is how well they extract and use

that data.

That’s what we at New Relic wanted to find out when it comes to data

about software. For the second consecutive year, New Relic surveyed

hundreds of companies in 22 different industries to check where they

stand on their journey to becoming data driven in their pursuit of

software success.

So what did we find out? Drumroll please… companies are getting

better at using data, but many admit they still have a good deal of

work ahead of them. Here are a few more highlights of our findings,

including notable changes from last year’s results:

• Software development overtook marketing as the department that respondents believe use data most effectively in 2015, with the majority of those responses coming from marketers themselves.

• Compared to 2014 when most companies released code monthly or even less frequently, more than half of companies released code weekly or more frequently in 2015.

• More respondents reported in 2015 that they didn’t know whether their company was better at using data than the competition (27% in 2015 versus 13.7% in 2014).

• Software teams reported that they are best at tracking application and infrastructure metrics, but not nearly as good at tracking customer experience.

• More than a quarter of companies believe their software teams are good at experimenting with new features.

• For three-fourths of respondents, cultural, technical, and other blockers persist in hindering experimentation critical to software innovation.

That’s not all of course. Read on for more juicy survey results and the

opportunity to compare your experience and your company’s to others

in your industry and situation.

1. “The internet of things and big data: Unlocking the power,” Charles McLellen, ZDNet, March 2, 2015.




http://www.zdnet.com/article/the-internet-of-things-and-big-data-unlocking-the-power/

Report

04



In December 2015, using Twitter and email to reach potential respondents,

New Relic conducted its second annual survey to find out how companies

are using data to effectively guide decisions about investment, product

development, and operations. The target audience for the survey was

people responsible for funding, designing, building, marketing, or operating

customer-facing Web or mobile software.

Nearly 500 people from 490 companies responded to the survey, with

an interesting mix of job types, industries, and company sizes:

• Customer-facing versus back-end/internal systems: The majority of respondents (72%) stated that they are responsible for success in delivering a mobile and/or Web customer experience. Another 16% of respondents work on back-end or internal systems and 12% were responsible for neither.

• All sizes of companies: Slightly more than half of the respondents (59%) work at companies with fewer than 100 employees, while 23% represented the mid-market where company size is between 100 and 1,000 employees. Large companies with more than 1,000 employees comprised 18% of respondents.

• Cross-section of industries: Overall, 39% of the survey respondents work for technology businesses. The single largest industry represented in the survey was B2B software at 13%, with Internet close behind at 12%.

• A variety of roles: More than a third of respondents were developers and software engineers (37%), while the second largest group was executives and managers in engineering and operations, with 23%. Rounding out the job areas were product management, marketing, sales, line of business, and security/compliance.

• Web-first versus historically off-line: Web-, mobile-, or software-native businesses represented 75% of the companies in the survey, while the remaining 25% self-identified as historically offline businesses with Web and/or mobile investments.

Finally, 65% of respondents identified themselves as New Relic

customers, with 80% of the New Relic customers being mobile/Web-

native companies and the remainder being traditional offline businesses.

Who we surveyed

72%

37% 39%

59%

work on mobile/Web customer experience

work at companies with <100 employees

are developers and software engineers

of respondents work for technology companies




Report

05



What does it take to become a data-driven company? Ask Etsy, the

online marketplace for handmade or vintage items, which has made

the transformation over the past five years. According to an article in

CIO magazine,2 data and metrics now support the entire company’s

operations. The motto at Etsy is, “If it moves, graph it.” After starting

with just five metrics in total, Etsy today has dozens of metrics for

each of its services. Measuring and tracking metrics allows Etsy to give

developers ownership of the success or failure of their process.

Is your company an Etsy in its use of data or is it still making its way

toward creating a data culture? Here’s what our findings show about

the state of data-driven software decisions.

Takeaway #1: Three-quarters of companies are on a data-driven journey When asked how their company uses data to make decisions on how

to invest, prioritize features, and run customer-facing Web and mobile

applications, nearly half of our respondents (47%) acknowledged that

they are still learning how to use data to drive important decisions.

Another 24% indicated that their organizations are making good use

of data in some places. Organizations making heavy use of data for

software decisions came in at 9%. Not surprisingly, 95% of that group

are Web- or mobile-native companies. To the dismay of 21% of respondents,

their companies are not even close to being able to use data to drive

investment or product decisions.

On the road to data-driven software success

2. “Why a data-driven transformation requires a cultural shift,” Thor Olavsrud, CIO magazine, April 6, 2015.

9%Data nerds are everywhere.

24%We are pretty good in a few places.

46%We are learning, but have a long way.

21%This question makes me sad. We’re not even close to doing that.

Companies’ level of maturity in using data to make software decisions.




http://www.cio.com/article/2905679/business-analytics/why-a-data-driven-transformation-requires-a-cultural-shift.html

Report

06



We also asked what would make the respondents’ departments

more effective in using data in 2016. More than half (52%) said that

dashboards that monitor key performance indicators would be the

number one thing that would help their departments become more

data-driven. The next three results on the data wish list included two

technology-specific items and a staffing need. All three were nearly

equal in importance for our respondents: an easier way to analyze data

(42%), data that’s more accessible to those who need it (41%), and

having more people with data analysis skills (39%).

Takeaway #2: Most businesses don’t feel they are better than the competition at using data We also asked how respondents believed they stacked up against the

competition when it comes to using data. Respondents were fairly evenly

split between feeling superior to or about the same as their competitors

(26% compared to 24%). Nearly a third (28%) didn’t know how their

companies compared to the competition. Interestingly, nearly 84% of

Web, mobile, or technology-native companies rated themselves ahead

of the competition compared to 16% of historically offline companies.

Large companies (those with 5,000 employees or more) reported

having more data nerds than competitors, while companies smaller

than 20 employees were more likely to report that they didn’t know

where they stand compared to the competition.

Industry also played a role in how companies felt about their own

data usage versus the competition. Companies in education,

telecommunication, real estate, and media and publishing were more

likely to report that they are superior to the competition when it

comes to using data. On the other hand, organizations in advertising

and marketing, government, retail and consumer goods, and B2B

technology were more likely to report that they were deficient in data

usage compared to their competitors.

52% Dashboards monitoring key performance indicators.

42% An easier way to analyze data.

41% Data that’s more accessible to those who need it.

39% More people with data analysis skills.

7% Nothing–we are pretty far behind.

4% Other.

What would help to be more effective with data.




Report

07



Speaking of competition, only 38% of the respondents believed that their

company is superior to competitors in using mobile and Web software

technologies to interact with customers. Less than half (40%) felt they were

on par with the competition, while 14% believed they were inferior.

Suggested action: Use data to prioritize

What can your team do to improve its use of data to drive software

decisions? One approach is to use data to prioritize projects and tasks,

so that people work on the areas that are most important to your

company’s goals. One way to do that is implement software analytics

that show, for example, which parts of your application are most

heavily used, which features are used by the customers who drive the

most revenue, or which areas of code generate the most errors.

38%Education

50%Tech/Telecom

50%Real Estate

32%Media & Publishing

Industries with companies reporting that they use data better than competitors.




http://newrelic.com/insights

Report

08



Takeaway #3: There’s a perception that the software development team uses data most effectively When asked which department at their company uses

data most effectively, 30% of respondents reported that

software development is the most data-savvy department,

followed by marketing at 21% and IT operations at 17%.

Respondents identifying themselves as product management

believe that both marketing and software development use

data equally well.

There was a certain amount of bias towards a respondent’s

own department, with people in IT operations, marketing, and

software development all reporting that their own departments

use data most effectively. For those respondents self-identified

as marketers, 60% named their own department as the most

effective user of data. Software development and IT operations

were a bit more modest, with each group citing their own

department as best data user 41% of the time.

30% Software Development

21% Marketing

17% IT Operations

8% None

7% Sales

7%

5%

3%1%1%

Finance

Other

Central IT

Human Resources

Manufacturing

Departments using data most effectively to make decisions.




Report

09



Takeaway #4: Companies are confident about their use of application metrics.This year we wanted to delve into specific areas of metrics to see how

companies ranked the ability to track and use them. We started with the

metrics we defined in our DevOps success framework (see our ebook on

the topic at http://newrelic.com/measuring-devops) and grouped them

into four categories:

• Infrastructure

• Application

• Customer experience

• Business results

Business ResultsInfrastructureApplication Customer Experience

1 2 3 4 5

Application InfrastructureBusiness Results Customer Experience

Well with visibility and collaboration across

the software team.

Decently within the org responsible.

Poorly within the org responsible.

Effectiveness of tracking each type of metric.

We asked respondents to rate how well they measure each type of metric

on a scale of 1 through 5, with 1 meaning they track it well with visibility

and collaboration across the software team and 5 meaning they track it

poorly. What we found was that of the four categories of metrics we asked

about, respondents reported that the software team tracks application

metrics more effectively than the other three types of metrics. The score

for application metrics was 2.81, while infrastructure was a close second,

with a score of 2.88. Business results and customer experience ranked

third and fourth respectively, with scores of 3.09 and 3.18.




http://newrelic.com/measuring-devops

Report

10



While not unusual, these findings definitely reflect the fact that most

companies have not yet reached data Shangri-La, and are still growing

their usage of data for customer experience and business decisions.

Similarly, we also asked respondents about data-driven feedback

loops for software teams in the categories of application performance,

customer experience, and business success. When asked which area

would have the most positive impact in helping their organization

deliver great customer-facing software in 2016, customer engagement

(36%) and application performance (35%) topped the list, with business

success close behind (29%).

For the purposes of the survey, customer engagement was defined

as a function delivered by product owners/managers and developers,

application performance by developers and IT operations, and business

success by customer behavior and marketing or sales teams.

Suggested action: Enable self-service feedback

People want feedback on the quality and impact of their work.

Giving engineers easy access to metrics that show performance,

adoption, and business results of the software they create can help

improve software quality. So start small with performance metrics

such as uptime, application response time, SQL query performance,

and resource usage. Then as your team gets used to a more data-

driven culture, introduce additional metrics that can help determine

the success of the team as a whole (for example, in proving the

effectiveness of DevOps efforts).





http://newrelic.com/measuring-devops




Report

11



Full speed ahead on innovation

One of the cornerstones of the agile development and DevOps movements

has been to increase the speed of product delivery, providing faster value

to customers and to the business. We were keen to see where companies

are today with release frequency and the ability to experiment that helps

create the groundwork for innovation.

Takeaway #5: One-quarter of software teams are good at experimenting with new features Experimenting helps companies evolve the software experiences they

deliver and enables the software development team to achieve better

results for the business. In our survey, 73% of survey respondents

indicated some type of difficulty that inhibits experimentation, while

27% said that their organizations are good at enabling experimentation.

The majority (73%) of those who said their organizations are good at

experimenting are New Relic customers.

More than a quarter of respondents (27%) reported that a limited ability

to measure what works and what doesn’t hindered experimentation.

Other reasons include: a culture that does not accept failures (13%), slow

software release frequency (10%), too difficult to roll back features (9%),

and lack of accountability between teams (8%).

27%

73%

say “We’re good at it!”

have some type of difficulty that inhibits experimentation.

Effectiveness of experimenting with new features.




Report

12



Takeaway #6: More than half of companies release code at least weeklyAccording to Puppet Labs’ 2015 State of DevOps Report, high-

performing IT organizations deploy 30 times more frequently with

200 times shorter lead times.3 Last year we found that a plurality of

companies (nearly 25%) in our survey released code to production

on a monthly basis. The next most common responses were quarterly

and then weekly deployments. A surprising 13% of respondents said

they didn’t release any code to production in 2014!

In 2015 however, more than half of the respondents (57%) said they

released code into production weekly or more frequently. One-quarter

of respondents reported releasing code monthly, with 10% releasing

code quarterly, and 9% even less frequently than that.

Weekly 27%Weekly17%

Monthly 24%Monthly25%

Multiple times a week 19%Daily7%

Multiple times a day 11%Multiple

times a day4%

Quarterly 10%Quarterly19%

Half year 4%Half year8%

Never 4%Never13%

Once 1%Once3%

2014

17% Weekly

4% Multiple times a day

25% Monthly

7% Daily

19% Quarterly

8% Half year

3% Once

13% Never

Weekly 27%Weekly17%

Monthly 24%Monthly25%

Multiple times a week 19%Daily7%

Multiple times a day 11%Multiple

times a day4%

Quarterly 10%Quarterly19%

Half year 4%Half year8%

Never 4%Never13%

Once 1%Once3%

2015

27% Weekly

11% Multiple times a day

24% Monthly

19% Multiple times a week

10% Quarterly

4% Half year

1% Once

4% Never

3. “Puppet Labs 2015 State of DevOps Report,” Puppet Labs, 2015.




https://puppetlabs.com/2015-devops-report

Report

13



For 2016, survey respondents are aiming to speed deployment even

more, with 66% expecting to deploy weekly or more frequently, an

increase of 11 percentage points compared to 2015.

Interestingly, 83% of New Relic customers said they deploy weekly,

daily, or more frequently. Compare that to 17% of non-New Relic

customers who said they deploy that often and you can see how

access to data provides the insight and confidence that enables rapid

iterations and deployment of new features.

Suggested action: Start tracking your speed

Teams adopting a DevOps approach typically move towards a

continuous delivery goal. To help your organization track its speed

of development and delivery, consider measuring your lead time

for changes and the frequency of code releases. At the same time,

quality is an equally important goal, so tracking mean time to

resolution can help you balance speed and quality goals.








Report

14



Job expectations now include a data cultureWe were right when we said everyone loves data. The vast majority of

respondents are excited about working in a data culture.

Takeaway #7: For nearly everyone, a data culture is a desire or expectation We asked survey respondents how they would feel about working at

a new job where the department they were going to join had a culture

of using data to prioritize work, build and adjust feedback loops in a

logical way, and hold team members and other groups accountable. A

whopping 90% indicated that they’d like or expect to work in a data-

driven environment. A small number (8%) were unsure, while 2% would

find it distasteful (we’re not sure what to make of that).

2014

46%

41%

11%2%

2015

47%

43%

8%

2%

Prospect of working for a company with a data-driven culture in 2014 and 2015.

No longer just “exciting,” most people expect a data-driven culture in today’s work environment.








Report

15



Takeaway #8: Let the Wookie win Finally, given last year’s release of the film, “Star Wars: The Force

Awakens,” we wanted to revisit on the question of which classic sci-fi

franchise our respondents preferred: Star Trek or Star Wars. The envelope

please: The majority of respondents favored the saga of the Force (65%)

compared to Star Trek’s final frontier. May the force (of data) be with you!

35%Beam me up, Scotty

65%Punch it, Chewie Star Wars or Star Trek?








Report

16



Looking forwardAs our survey results make clear, companies are continuing on their journey to a data culture, making significant gains in areas such as time to market.

Here at New Relic, we fully expect to see that and other data-driven trends continue, with even more companies empowering teams with meaningful

data for more informed decisions, faster innovation, and greater responsiveness to customers.

For all the data nerds, wannabes, and even skeptics reading this report, we hope you enjoyed seeing this data as much as we enjoyed bringing it to you.

After all, we’re the company that collects more than 690 billion metrics every day so you can go out and make informed decisions about your software.

To learn more about using data to make better software decisions, visit: http://newrelic.com/software-analytics.

About New RelicNew Relic is a software analytics company that delivers real-time insights to hundreds of thousands of users and tens of thousands of paid business accounts. As a multi-tenant

SaaS platform, the New Relic Software Analytics Cloud helps companies securely monitor their production software in virtually any environment, without having to build or

maintain dedicated infrastructure. New Relic helps companies improve application performance, create delightful customer experiences, and realize business success. Learn

more at newrelic.com.

New Relic, San Francisco HQ188 Spear Street, Suite 1200

San Francisco, CA 94105

New Relic, Portland111 SW 5th Avenue, Suite 2800

Portland, OR 97204

Tel: [email protected]

www.newrelic.com

New Relic, Dublin34-39 Nassau Street, 3rd Floor

Dublin 2, Ireland




http://newrelic.com/software-analytics

http://newrelic.com

mailto:support%40newrelic.com?subject=


survey report a deep dive into data culture · respondents believe use data most effectively in...

Documents