practical advice to build a data driven company

50 AVENUE DES CHAMPS-ÉLYSÉES 75008 PARIS > FRANCE > WWW.OCTO.COM

HADOOP SUMMIT 2016 - DUBLIN

PRACTICAL ADVICE TO BUILD A DATA DRIVEN COMPANY

Simon MABY@simonmaby

2OCTO TECHNOLOGY > THERE IS A BETTER WAY

Story : Data Driven E-Commerce

3

A continuous improvement of all business processes, through a smart use of the data, all the

time, everywhere and to all purposes

OCTO TECHNOLOGY > THERE IS A BETTER WAY

4

BEING DATA DRIVEN IS BEING LEAN


IDEA

CODEDATA

BUILD

MEASURE

LEARN

5

REQUIREMENTS


IDEA

CODE

DATA Data must be easily accessible

Business must be aware of opportunities to use algorithms

Datascience projects should have the lowest time to market possible

6

DATA

7

DATAData must be easily accessible


8

Your Datalake is a service to your company. It should be managed like a startup

Your employees are you first clients. The more they use it, the more you are Data Driven


9

FOCUS ON USABILITY OVER ARCHITECTURE


Services

Datalake

Datalake Team :OPS - DEVs - DESIGNERS

End Users and projects

Design services for usability and grant support

Gather requirements

and usage metrics

10

FOCUS ON USABILITY OVER ARCHITECTURE : EXAMPLES

How simple is it to share data to other projects?

How simple is it to suscribe to a data feed?

Is it possible to run a full search on available datasets?

Is it possible to ask other projects for details about their data through a social network?

Auto-completion over SQL request from other projects?

Bookmarking, sharing, upvoting datasets, tagging metadata…OCTO TECHNOLOGY > THERE IS A BETTER WAY

11

CODE

12

CODEDatascience projects should have the lowest time

to market possible


13

EXPLORATION VERSUS PREDICTION


Explore as quickly as possible

Deliver frequently in production


(Not so) Big Data Infrastructure(For exploration)

15

WHAT IF WE GIVE LESS DATA TO OUR ALGORITHMS?


Cf. Zoltan Prekopcsak, Hadoop Summit EU. 2015

https://prezi.com/user/preko/

https://prezi.com/user/preko/

16

FEATURE TEAMS TO DELIVER CODE READY FOR PRODUCTION


Business rep.

Developer

Data Sc.

17

MESSAGE BROKER TO REUSE DATA FLOWS


App A App B

DWDB X

App A App B

DW DB X

Kafka

App C

? ? ?- Custom dev- Data formats?- SLA?- Scheduling?…

- Standard format- Prod Ready- Exploration and prod will share same formats

18

KAPPA ARCHITECTURE : EVERYTHING IS A STREAM


Stream Data Stream Processing Serving DB

Topic Streaming app v1

Streaming app v2

Result data v1

Result data v2Kafka

Batch jobs are just historical data you send into a streaming app Application code is decoupled from technical requirements One shot exploration code respecting the stream abstraction can go in

production easily

19

IDEAS

20

IDEASBusiness must be aware of the opportunities to

use algorithms


21

MIX THESE PEOPLE


BusinessKnows what is

valuable

Data ScientistKnows what is

feasible

Culture &Collaboration

22

FEATURE TEAMS ONCE AGAIN


Business rep.

Developer

Data Sc.

23

EXPLAIN THEM THAT MACHINE LEARNING IS EASY (IT’S METHODOLOGY)


24

EXPLAIN THEM THAT MACHINE LEARNING IS EASY (IT’S MAGIC)


25

SPEND TIME TOGETHER

Show them the data

Pair Programming

Swap roles for one day


26

SOFTWARE IS EATING THE WORLD : MAKE THEM CODE


Story : Octo Datascience Competition Platform

HOW WIDELY DATADRIVEN IS YOUR COMPANY?

Everybody is willing to make value out of the available data

Data serves not only the core business but every single function

Data is used in day-to-day activity in real-time


HOW DEEPLY DATADRIVEN IS YOUR COMPANY?


You are using cutting edges algorithms to automate processes

You are used to A/B testing based on data every week

You cross multiple data sources to build insights and models

practical advice to build a data driven company

Technology