2018 trend report: enterprise ai adoption

AI PROMISE RUNS INTO ENTERPRISE REALITY 1 of 10

eBook

2018 TREND REPORT: Enterprise AI AdoptionHow today’s largest companies are overcoming the top challenges of AI.

SPONSORED CONTENT

2018 TREND REPORT: ENTERPRISE AI ADOPTION 2 of 10

Executive SummaryEnterprises are making significant investments in artificial intelligence (AI) technology as they attempt to retool

their business and create competitive advantage. This CIO survey of global data science and engineering leaders

across multiple industries found that almost 90% of them are making significant AI investments, but very few are

realizing the full benefits of their investments.

Only 1 in 3 AI projects are successful and it takes more than 6 months to go from concept to production, with a

significant portion of them never making it to production — creating an AI dilemma for organizations.

The very thing that makes AI possible is also making it challenging to implement: data. About 96% of organizations

say data-related challenges are the most common obstacle when moving AI projects to production. Enterprise

data is not AI-enabled and is siloed across hundreds of systems such as data warehouses, data lakes, databases

and file systems. And machine learning (ML) frameworks such as TensorFlow and others don’t do data processing.

Since data systems don’t “do AI” and these AI technologies don’t “do data,” organizations end up using on

average 7 disparate tools which create friction and slow down projects. To make matters worse, the survey found

that 80% of them face collaboration challenges as data science and engineering teams are in organizational silos.

So, what will help these organizations conquer the AI dilemma? According to the survey, 90% of the respondents

believe that unified analytics — the approach of unifying data processing with ML frameworks and facilitating

data science and engineering collaboration across the ML lifecycle, will conquer the AI dilemma. Unified Analytics

is a new category of solutions that unify data science and engineering, making AI much more achievable for

organizations. Unified Analytics makes it easier for data engineers to build data pipelines across siloed systems

and prepare labeled datasets for model building while enabling data scientists to explore and visualize data and

build models collaboratively.

The Databricks Unified Analytics Platform, from the original creators of Apache Spark™, unifies data science

and engineering across the ML lifecycle from data preparation to experimentation and deployment of ML

applications — enabling companies to accelerate innovation with AI. The Databricks platform provides one

engine to prepare high quality data at massive scale and iteratively train ML models on the same data while

leveraging all the popular open source frameworks. It also provides collaboration capabilities for data scientists

and engineers to work effectively across the entire AI lifecycle. Organizations that succeed in unifying their

data at scale with the best AI technologies will have a significantly higher chance of success with AI.

TOC3 Introduction

3 AI Dilemma

4 Data Related Challenges

6 Complexity from Explosion of ML Frameworks

7 Data Science and Engineering Silos

8 A New Category — Unified Analytics

9 Databricks Unified Analytics Platform


IntroductionCIO/IDG Research Services surveyed more than 200 IT executives at larger companies (1,000+ employees) in the

U.S. and Europe who are either considering or using AI technology. We wanted to understand the AI investments,

expected business outcomes, challenges, and the drivers of AI success across the landscape.

AI Dilemma — Nearly 90% investing in AI, very few succeedingIt’s clear that respondents are putting significant resources into their AI projects in hopes of forging new business

models that take advantage of data and ML across industries including the discovery of new life saving drugs,

detecting fraudulent and malicious behavior, improving global supply chain management, and creating a highly

personalized digital experience for their customers.

Despite the challenges, survey respondents are pursuing AI with gusto. It should be no surprise, then, that two-

thirds of respondents expect their AI investments to increase in the coming year. (See graphic below.)

Only 1 in 3 AI projects are

successful and it takes more than

6 months to go from development

to production.

Use/Planned Use of AI Technology Business Benefits Experienced from AI Projects

Predictive analytics

IT automation

Customer analytics

Security, fraud analysis and investigation

IoT analytics

Risk assessment

Improved security

Improved customer experience

Increased innovation

Better quality/more effective decision-making

Increased competitive advantage

Product/service transformation Source: IDG Research

48%

45%

44%

33%

27%

27%

29%

29%

26%

26%

23%

23%


“AI has massive potential to drive disruptive innovations affecting most enterprises on the planet. It’s pervasive

across all industries. It is used in genomics to accelerate drug discovery and drive personalized medicine. It

is being applied to manufacturing to improve operational efficiencies of product development and delivery

processes,” Bharath Gowda, VP of Product Marketing at Databricks, says. “In spite of the enormous potential,

very few companies are being successful with scaling with AI efforts”

Data-related Challenges Are Hindering 96% of Organizations from Achieving AIBut the CIO/IDG survey shows the full benefits of AI are not yet being realized for a variety of reasons, but with

one overarching theme: Data. Nearly all respondents (96%) cited multiple data-related challenges when it comes

time to move projects to production (see graphic below).

“ Simple models and a lot of data trump more elaborate models based

on less data.”PETER NORVIG

Research Director at Google

Challenges of Moving from Concept to Production

Preparation and aggregation of large datasets in a

timely fashion for analytics

Data exploration and iterative model training

with largesets of data

Deployment of models to production quickly

and reliably

Source: IDG ResearchVery/Extremely Challenging

Not very challenging

Somewhat challenging

1%56% 42%

4%56% 40%

10%53% 37%


“ Sifting through massive amounts of data to

identify useful signals is an enormous computational challenge; it’s the type of dataset and computation that DevOps nightmares

and Data Science dreams are made of.”

CHRIS ROBISON Lead Data Scientist at Overstock.com

And the data silos far outpace the other issues when talking about the data-related challenges, with technology

complexity also creating the second biggest challenge.

Data-Related Challenges to Move AI Projects to Production

Data silos (in different parts of the business, acrossdifferent locations, etc.)

Too many technologies in place/technology complexity

Accessing large sets of clean data quickly

Difficult for those processing/preparing data and thosecreating the data models to collaborate

Lack of ample access to data engineering and datascience talent to make AI a reality

Difficult for data scientists with varying skills andtechnology knowledge to collaborate

Lack of a scalable, reliable technology platform toprocess large data sets

51%

37%

35%

35%

29%

25%

24% Source: IDG Research

Gowda says, “For data scientists, it’s been proven that simple models built from large amounts of data produce

better results than very sophisticated models built from small sets of data,” he says. “So, more data means better

models — data is the fuel that powers AI. Clean, reliable data that is accessible to data scientists is the key to

success. Therein lies the challenge for enterprises — transforming siloed messy data into clean labeled data for

model development”


Increasing Complexity: Organizations Invest in an Average of Seven Different ML ToolsThe differences between the data engineering and data science teams also extend to the tools they use, and

there are many.

The vast majority (87%) invest in various sorts of data and AI related technologies to help with data preparation,

exploration, and modeling, including:

85% Data processing tools such as Apache Spark, Hadoop/MapReduce, and Google BigQuery,

used by 85% of respondents.

65% Data Streaming tools such as Flume, Kafka, and Onyx

80% Machine learning tools such as Azure ML, Amazon ML, and Spark MLlib

65% Deep learning tools such as Google TensorFlow, Microsoft CNTK, and Deeplearning4j (DL4J)

Overall, survey results show that organizations are using an average of seven different machine learning and deep

learning tools and frameworks, creating a highly complex environment that can slow efficiencies.

“To derive value from AI, enterprises are dependent on their existing data and ability to iteratively do ML on

massive datasets. Today’s data engineers and data scientists use numerous, disconnected tools to accomplish

this, including a zoo of ML frameworks,” Gowda says.

Click Streams

...

Video/ Speech

SensorData (IoT)

Emails/Web Pages

Customer Data

Great for Data, but not AI Great for AI, but not data

Click Streams

...

Video/ Speech

SensorData (IoT)

Emails/Web Pages

Customer Data

Great for Data, but not AI Great for AI, but not data

x

Divide Between Data & AI Technologies


Siloed Data Science and Engineering Teams: 80% Experience Reduced Productivity as a Result Technology skills, leadership, and lack of a cohesive strategy are the biggest hurdles faced by data engineering

and data science. (See graphic below.)

Collaboration between data engineering and data scientists

Extremely challenging

Not very challenging

Very challenging

Somewhat challenging

Source: IDG Research

9%

31%

40%

18%

2%Not at all challenging

Challenges for Data Engineering and Science

Technology skills/capabilities

gaps

Lack of project oversight/ leadership

Lack of a unified strategy

Disagreements regarding data

ownership/control

Limited understanding

of roles and responsibilities outside of one’s

own team

55%

47%

28%

34%

49%43%

30%

37%

28%24%

Source: IDG Research

Data Engineering

Data Science

“Disjointed development and data science teams

is a major obstacle in successfully doing data

analytics.”SAMAN MICHAEL FAR

Senior Vice President of Technology at FINRA


Unified Analytics — Many Need A New Category of Solutions to Conquer AI DilemmaSurvey respondents are clear that they would welcome such tools. Nearly 4 out of 5 (79%) said an end-to-end

analytics platform that unified big data and AI, while fostering better collaboration between data engineering

and data science teams, would be highly valuable.

Other features that would be welcome in such a platform include:

High quality performance for large data sets.

Built-in integration with various data sources.

Collaborative spaces that allow data scientists with different skills to work together.

Cloud-native platform to enable elastic scalability.

Built-in data management capability for building large data pipelines.

Support for multiple clouds.

Apache Spark was the first Unified

Analytics engine to unify data (data

engineering) with AI (data science).

Apache Spark has become the

de-facto data processing and AI

engine in enterprises today due to its

speed, ease of use, and sophisticated

analytics. Spark simplifies data

preparation for AI by unifying data at

massive scale across various sources

— cloud storage, file systems, key-

value stores, and message buses.

Spark also unifies data and AI with a

consistent set of APIs for simple data

loading, batch/stream processing,

SQL Analytics, Stream Analytics, and

Machine Learning.


Databricks Unified Analytics PlatformDatabricks accelerates innovation by unifying data science, engineering, and business. Through a fully managed,

cloud-based service built by the original creators of Apache Spark, the Databricks Unified Analytics Platform

lowers the barrier for enterprises to innovate with AI and accelerates their innovation.

“Databricks lets us focus on business problems and make data science very simple.”

DAN MORRIS Senior Director of Product Analytics at broadcast

giant Viacom, which used Databricks to help it identify video quality issues, increase customer loyalty,

and improve advertising performance.

DATABRICKS CLOUD SERVICE

DATABRICKS WORKSPACE

DATABRICKS RUNTIME

Reliable & Scalable Simple & Integrated

End to end ML lifecycle

Dashboards

Jobs

UNIFIED ANALYTICS PLATFORM

Databricks Delta ML Frameworks

API's

Notebooks

Models

Data EngineeringSpeed up the preparation of high quality data, essential for best-in-class ML applications, at scale.

Data ScienceCollaboratively explore large datasets, build models iteratively and deploy across multiple platforms.


Databricks Workspace — Unify data science and engineering teams

The Databricks Workspace empowers data science and engineering to collaborate using interactive notebooks

that are tightly integrated with cloud-native Apache SparkTM clusters. Real-time collaboration capability and

the ability to program in multiple languages increases data science productivity significantly. in the notebooks

increases the It integrates with MLflow, an open source, cross-cloud framework that tracks experiments and

enables deployment across multiples clouds dramatically simplify the ML workflow.

Databricks Runtime — Unify data and machine learning at massive scale

Databricks Runtime allows engineers to quickly build data pipelines at massive scale by using Delta tables which

bring data reliability and performance optimizations to data lakes. Using reliable data in Delta tables, data scien-

tists can continuously train and deploy state-of-the-art ML models by using pre-configured clusters that include

most of the popular ML frameworks such as TensorFlow, Horovod, Keras, XGBoost, and scikit-learn.

Databricks Cloud Service with enterprise-grade security

Databricks automates and simplifies dev-ops by abstracting the complexity of the data infrastructure by auto-

configuring and auto-scaling clusters; and provides enterprise-grade security and compliance, along with best-

in-class Spark support from the original creators of Apache Spark. Databricks Unified Analytics Platform provides

enterprise-grade security with encryption, auditing, role-based control and HIPAA and GDPR compliance.

“Enterprises are rightly pursuing multiple AI projects as they seek to realize the business benefits the technology

can provide. Data science is going to change the world,” as Databricks’ Gowda says. But as the CIO/IDG survey

makes clear, enterprises are often falling short of success in large part because they’re spending too much time

massaging data and struggling to collaborate between teams. The Databricks Unified Analytics Platform can

likewise help your company take advantage of AI technology to address strategic business objectives. It’ll help

you meet data and collaboration challenges and get AI projects out the door on time and on budget.

To learn more, visit: www.databricks.com

IDG Communications, Inc.

SPONSORED CONTENT

Benefits of AI are Streaming into Viacom Viacom turned to AI technology to help it analyze peta-bytes of network data to improve the performance of its network, grow its audience and improve the advertising performance of its 170 cable, broadcast and online networks in some 160 countries.

Viacom built a real-time analytics solution using the Databricks Unified Analytics Platform to constantly monitor the quality of video feeds and reallocate resources as necessary to ensure best-in-class customer experience. Databricks Runtime has the horsepower to keep up with Viacom’s constant flow of streaming data. And the Databricks Collaborative Workspace enables Viacom data scientists and engi-neering groups to collaborate with each other and with the business.

The results speak for themselves:

With the ability to predict video trends and issues, Viacom has reduced video start delays by 33%.

Leveraging customer data, Viacom has increased viewer retention 3.5 to 7 times

The ability to target customers with personalized ads based on comScore ratings and viewing behavior has improved ad conversion rates.

Dan Morris, Senior Director of Product Analytics for Viacom, isn’t done; rather, he’s looking for more ways to apply the technology. “Now it’s a question of how we bring these benefits to others in the organization who might not be aware of what they can do with this type of platform,” he says.

https://databricks.com/customers/viacom

2018 trend report: enterprise ai adoption

Documents