analyst first at the chief data officer forum melbourne, august 2015

Effec%ve Analy%cs Leadership -‐ What Every Execu%ve Must Know

Dr. Eugene Dubossarsky

Principal Founder : Analyst First Director : Presciient

Convener: Data Science Sydney and Sydney Users of R Forum [email protected]

+61414573322 @cargomoose

Presciient and Analyst First

presciient.com

•  For upcoming courses on data science, analyKcs, R, visualisaKon, fraud detecKon, soL skills in analyKcs, managing analyKcs and other good things.

hNp://analysPirst.com/analyst-‐first-‐101/ hNp://analysPirst.com/core-‐principles/ for more thinking along the lines of this presentaKon.

Road Map

•  1. DefiniKons, Hard Truths, Hard QuesKons and MoKvaKon

•  2. AnalyKcs Sponsorship and the AnalyKcs FuncKon – Good and Bad

•  3. AnalyKcs Sponsorship : How to get it (More) right

•  4. AnalyKcs Skills and Training •  5. Hiring a CDO and AnalyKcs Team

Road Map

•  1. Defini%ons, Hard Truths, Hard Ques%ons and Mo%va%on




A DefiniKon

•  AnalyKcs is the use of data to support business decision making. –  This may involve complex staKsKcal, computaKonal and visual analysis of data.

But, Conversely : •  if it doesn’t support decision making it isn’t really analyKcs. – Unfortunately, this does not rule out the complex staKsKcal, computaKonal and visual analysis of data…

Some Hard Truths

“The same enterprises that seem most confused about Big Data seem to be the ones launching Big Data projects. What gives?” “According to a recent Gartner report, 64% of enterprises surveyed indicate that they're deploying or planning Big Data projects. Yet even more acknowledge that they sKll don't know what to do with Big Data. Have the inmates officially taken over the Big Data asylum?” Gartner On Big Data: “Everyone's Doing It, No One Knows Why” 18/09/2013

Some Hard Truths

In a difficult economic environment: organisaKons are less likely to pay for something they don’t value and even less for something they don’t understand.

Some Hard QuesKons

•  In your organisaKon : – What would happen if the analyKcs funcKon disappeared tomorrow ?

–  In an economic downturn, would your analyKcs budget go up or down ?

– What is your CDO really worth to the company ? –  Is your CDO a contender for CEO ? –  Is analyKcs really vital to top decision makers for make key decisions ?

“Here Be Dragons”

•  Does your business really need data analy%cs ? (Even if it says it does. And I did say the business. I didn’t say your career)

•  Is data analy%cs something you really want to do ? (even if it looks good on your resume. the consequences may not be what you think they are.)

•  Is this something you are really ready for ? (AnalyKcs is probably not what you think it is. Even if you already work there. Especially then)

Why This Stuff MaNers

•  If we hit another economic crisis :

Will analyKcs rise in prominence (essenKal to good decision making, vital source of ongoing compeKKve advantage) or disappear (discreKonary expense / poliKcal football that nobody really understands and less appreciate) ?

Road Map


•  2. Analy%cs Sponsorship and the Analy%cs Func%on – Good and Bad



Reasons to Sponsor an AnalyKcs FuncKon -‐ The Good, The Bad and the

Ugly •  CompeKtors, DisrupKon uncertainty : need to make

beNer strategic decisions. Or else. •  As above, we also need more efficient operaKons / beNer operaKonal decisions.

•  Because we were told to •  Because it makes us look good •  Because that’s the Job DescripKon •  We need to generate beNer numbers to make compeKtors / regulators / other stakeholders Go Away.

Reasons to Sponsor an AnalyKcs FuncKon -‐ The Good

•  Compe%tors, Disrup%on uncertainty : we need to make be^er strategic decisions.

•  As above, we also need more efficient opera%ons / be^er opera%onal decisions. How oLen do we see this ? In what industries ? What organisaKons have no choice but to be like this ?

Reasons to Sponsor an AnalyKcs FuncKon -‐ The Bad

•  “Because we were told to” •  “Because it makes us look good” •  “Because that’s the Job Descrip%on” •  “It’s cubng edge/best prac%ce/everybody else seems to be doing it”

This is most advanced analyKcs funcKons in large orgs. Most people in this situaKon don’t see the problem. What creates, sustains this state of affairs ? What can change it ?

Reasons to Sponsor an AnalyKcs FuncKon -‐ the Ugly

•  We need to generate be^er numbers to make compe%tors / regulators / other stakeholders Go Away. – This is most BI funcKons. – This is NOT Decision support.

“AcKonable Insights” vs AcKonable Insights

•  Yes, insights need to be acKonable to be valuable.

•  No, “acKonable” does not mean “cut-‐and-‐dried decision, no thinking required”.

•  This is oLen the understood meaning. •  Bonus quesKon : what is the actual job of highly paid decision makers who do not view the digesKon of insights (we call it “thinking”) part of their job ?

The Ideal AnalyKcs Sponsor (AKA the CDO’s BOSS)

–  VERY senior, and maybe next in line for CEO if not there already. –  has clear, well defined goals and expectaKons. –  demands analyKcs insights for decision support (NOT “acKonable insights”)

and disKnguishes good insights from poor. –  Understands, values and seeks improvement in key metrics (such as predicKve

accuracy and related value/risk measures) arising from predicKve analyKcs. QuesKons and improves the relevance of those metrics. Actually understands them !

–  Supports the analyKcs funcKon appropriately in terms of tools, data, talent, execuKon and poliKcal cover

–  Welcomes their own job changing –  Is indifferent to the sufferings of those made uncomfortable by data –  is the key CUSTOMER of analy%cs –  uses analy%cs to make decisions. –  Has “Skin in the game” –  Needs analy%cs for an edge against smart, ruthless compe%tors –  Wants to win against EXTERNAL compe%tors –  Increases the analy%cs budget in %mes of crisis – good decision making,

compe%%ve edge ma^er more !

The Ideal AnalyKcs FuncKon –  Exists to support (but not make!) decisions. –  Provides insights to decision makers –  Delivers relevantly measurable value from operaKonal analyKcs, parKcularly

predicKve modelling. –  Is appreciated by decision makers for decision support and measurable

operaKonal value improvement. –  Receives appropriate support to deliver more value –  Does something new every day –  Is sponsored by its customers, and managed by people who understand the

KPIs (hard and soL) it delivers. –  Delivers high mulKples of its cost –  Keeps a low profile. Doesn’t self-‐promote much. –  Is an Intelligence FuncKon –  Grows –  Transforms the company –  Has “skin in the game” –  Makes some people unhappy

But, OLen •  The Sponsor:

–  Not truly C-‐ Suite. (what does that make the “C”DO ?) –  Pays the bills but isn’t a decision maker. –  Is looking for someone else, an actual or purported decision maker or even someone more

junior to noKce, appreciate and support the analyKcs funcKon – sales ! –  Makes demands on the analyKcs funcKon for things that look flashy, ideally visual. Not real

insights or decision support for actual decision makers. –  Has no idea what the team does or why, other than to produce “a number” occasionally,

usually to make someone else happy and with no appreciaKon of the accuracy of that number.

–  Dismisses the above issue as “technical” –  Has no idea how to support the analyKcs funcKon. –  Cares more about internal poliKcs than external compeKKon –  Isn’t a CUSTOMER. More like a temporary owner / reseller. –  Has no “skin in the game” –  Making decisions isn’t really part of their job –  Neither is thinking about analy%cs results. –  LOOOOVES “Ac%onable insights”, flashy presenta%ons, brand names and buzzwords. –  Cuts the analy%cs budget it %mes of crisis. It is a poorly understood discre%onary expense.

But, OLen: •  The AnalyKcs Team

–  Formed to fulfill acKon item “form an analyKcs team” –  Commissioned by someone who is NOT a customer of the funcKon

and pays aNenKon to something else enKrely. –  Is constantly looking for someone to support their work. –  Is overworked and low morale – too many specialists doing menial

work, not enough data wranglers to support them –  Works in reacKve panic mode, delivers “the numbers” on demand –

mostly for compliance or poliKcal reasons – not decision support. –  Is constantly promoted / sold within the company and externally at

conferences. –  Is managed by someone who does not understand the KPIs of their

work, nor how to support it –  Is managed according to determinisKc / waterfall methods. –  Could disappear tomorrow. Would anybody noKce ?

Road Map



•  3. Analy%cs Sponsorship : How to get it (More) right


Approaching The Ideal – Nice To Have •  Find an organisaKon, or at least business funcKon that actually needs analyKcs to survive and thrive. Preferably one with real compeKtors, and no assurance it will be around tomorrow.

•  Find a sponsor (maybe more than one) that actually makes decisions, and wants to make beNer ones, and has the clout to supply and protect the funcKon, as well as be its best customer.

•  A bit like “to be a successful trader : buy low and sell high”

Approaching The Ideal – Intelligence FuncKon

•  Be a secreKve, value-‐adding team, reporKng discretely to the sponsor(s), and in constant contact with them as trusted advisers.

•  Report discreetly. •  Stay under the radar. Let the sponsor shine.

Approaching The Ideal -‐ Agility •  Try many things. Focus on the ones that work.

Allow many to fail. Learn from them all. Keep them off the radar unless they succeed.

•  Keep your budget lean. You don’t have to buy soLware. Some of the best stuff is free.

•  The less “stakeholders” and dependencies the beNer.

•  Stay lean. Avoid large, ill-‐defined expectaKons.

Approaching The Ideal – Human Infrastructure

•  Focus on good people, skills, experience. •  Get quality people. •  Get quality training •  Get mentoring / advice / guidance •  You can’t buy experience, you have to earn it.

Approaching The Ideal – Human Infrastructure

•  Get more data wranglers. Most people don’t have enough.

•  Have subject maNer experts. CommunicaKon challenge is theirs as much as the data people’s. Get them data literate.

•  Three broad disciplines : Subject MaNer Experts, IT / Engineers/ Data Wranglers, Data ScienKsts.

•  Two disciplines in the same head are gold. Three in the same head is extremely rare.

•  Ideally whole team has minimal literacy in all three. •  Data ScienKsts will absorb the other two more easily than non-‐data-‐scienKsts will absorb data science.

Road Map




•  4. Analy%cs Skills and Training •  5. Hiring a CDO and AnalyKcs Team

The Most Important Talent in AnalyKcs

•  The CDO’s Boss! –  The most important talent in analyKcs is the decision making / insights ingesKon talent of the customers of analyKcs.

–  The second most important talent is the ability to create, sustain, support and grow and analyKcs team.

– Needs a high level of literacy to be an effecKve user, customer, criKc, manager, supporter of analyKcs.

–  Is there any control for that ? –  Strangely ignored in most CDO/Data ScienKst recruiKng efforts…

Good AnalyKcs People And Their CommunicaKon Skills

An aside: AnalyKcs people with good communicaKon and business skills are valuable and rare. There are someKmes problems with analyKcs people’s communicaKon skills. Most analyKcs people are however painfully aware of these issues and work hard to correct them. They usually feel like the problem is enKrely their fault, when…


Much of the problem is sponsor/manager/stakeholder/”business” inability to “handle the truth”, either to accept poliKcally uncomfortable truths or process inherently complex ones. Business people with actual good communicaKon skills and actual good business skills are also rare. Communica%on is a two way street.


Being liked, accepted and admired by other business people is, surprisingly, not always the most vital business skill. CommunicaKng complex issues correctly and not oversimplifying or missing the point -‐ is a business skill. “Making decisions with complex informa%on under uncertainty” – is a business skill. Perhaps THE business skill.

Skills And Training •  Basic Data Literacy •  Reasoning : Logic, Science and Probability •  Coding (R, Python, etc) •  RelaKonal Reasoning •  Data VisualisaKon (for analysis and communicaKon) •  PredicKve Modelling (for predicKons and insights) •  Scalable tools (Hadoop, Spark, Cloud plaPorms etc) •  ForecasKng •  SimulaKon •  Networks •  Text •  OpKmisaKon •  Managing under uncertainty (Agile, Cynefin, OODA, Analyst First)

“Technical” vs. “Strategic”

•  The boundary isn’t where you might think •  Logic and systems thinking – core competencies of decision making are apparently “technical”

•  So is the ability to ingest complex informaKon in order to make effecKve strategic decisions.

•  Basic staKsKcal literacy, the scienKfic method – you can’t make strategic decisions off analyKcs without them. –  The future is a probability distribuKon. –  CorrelaKon does not imply causaKon. But if you don’t understand either one, why are you managing an analyKcs funcKon ?

AnalyKcs Literacy (for sponsors and CDOs)

•  If AnalyKcs was a restaurant : – You don’t need to be a chef, – but you need to know the basic rules of a restaurant: you need to be able to read the menu, order, cut, chew and swallow.

– You also need to be a connoisseur – You need to keep the kitchen supplied and in business.

ExisKng ExecuKve Literacies

•  Simple literacy and numeracy •  Financial literacy •  Computer literacy •  Process and Project Literacy •  Spreadsheets and tabular data, pie and bar charts

•  Logic

The New ExecuKve Literacies

•  Logic •  ProbabilisKc reasoning •  Common cogniKve biases •  The scienKfic method / experimentaKon /causality •  Visual and relaKonal data •  The basics of data science •  ForecasKng and Decision Making •  Non-‐determinisKc management •  Decision Making From Data Under Complexity and Uncertainty

Managing AnalyKcs : Determinism vs Reality

•  Deliverables of analyKcs (findings ! Insights ! Model accuracy !) are not determined prior to analysis.

•  Further tasks arising from findings. They can’t be idenKfied ahead of Kme. So can’t really use waterfall approaches / convenKonal IT management.

•  Analy%cs is NOT IT ! -‐ Analysts are not developers.

Determinism vs Reality

•  Analysts do something new every day •  AnalyKcs done right is more like military intelligence.

•  The right way to Manage Analy%cs : –  truly Agile methods. (true to the Agile Manifesto) – Try many things, expec%ng most to fail – Working closely with decision maker / sponsor as discreet advisor

The Technological Trap •  Analy%cs is actually cheaper, easier and faster than some people might

want you to think. That’s why ooen small startups can manage it where large companies can’t. Focusing on the technology you lose sight of this too easily.

•  AnalyKcs need good IT, the way Olympic runners need good shoes •  The runner should be the focus, not the shoes. •  A poor runner is the world’s best shoes is no match for the best one in

barely adequate shoes. Or even barefoot. •  If athle%c running was like analy%cs, most of the focus would be on

running shoes, and so li^le on runners, and even less on coaches and judges. There would be very few races, but lots of expensive shoes bought, and much %me spent on 5 year “journeys” to building running tracks nobody remembers how to use when they are ready.

Hiring A CDO – What do people cost ?

•  How much would you pay for a Chief Data Officer ?

•  Why ? •  What is the limit ? •  If the limit is “Market rate” : does it gel with the hype ?

•  Whose job is it to know and assess this ? •  What is the real purpose of hiring a CDO ? •  Does anyone even know ?

Road Map




•  4. AnalyKcs Skills and Training •  5. Hiring a CDO and Building an Analy%cs Team

Hiring Good AnalyKcs People Apparently “There isn’t enough talent in Big Data / Data Science / AnalyKcs” But: Do recruiters / in-‐house IT / THE BUSINESS even know good from bad ? Is “good” in general the same as “fit for purpose” ? Does business know what to do with good people if they find them ? There are issues with recruitment. They are only symptoms of issues with the buy side.

Hiring A CDO

•  Hire from the top down. •  Hire someone GOOD. Ask other GOOD people what that means – don’t just rely on recruiters and IT.

•  Only other good people really know who is good.

•  Let the new CDO hire their own team, and build their own tool set and infrastructure.

Hiring People

•  Hire LOTS of data wranglers. •  One or two for every data scienKst / analyst. •  Hire these first, or right aLer the (Good) CDO. •  Otherwise, talent is wasted. •  Don’t overload your seniors with wrangler work. It’s a waste of their rarer skills. It may also not be what they are good at !

Building Capability

•  Hire carefully. •  Hire slowly. •  Know what you are hiring for. •  Leave the hiring to a good CDO… •  Only buy sooware / hardware you absolutely know that you need. Make do with free/open source wherever possible. It’s ooen be^er, and close to industry standard

Blowback •  AnalyKcs is ONLY useful for EXTERNAL COMPETITIVE edge. •  It can be USELESS or HARMFUL to internal compeKKon and poliKcs, especially

where there is no significant external compeKKve pressure. •  AnalyKcs does NOT make people’s jobs easier. •  Some people are right to be mistrusPul of analyKcs – it makes them accountable

and obsolete, while making their jobs harder. •  AnalyKcs is not there to make jobs easier, careers more brilliant or employees

happier. It is there to WIN. Winning can be a brutal zero-‐sum game. •  Change is not easy for most people. •  AnalyKcs encourages a measurable, empirical, meritocraKc environment – does

this play to the strengths and preferences of the management class in your organisaKon ?

•  Successful analyKcs is ruthlessly transformaKve, and cares nothing for poliKcs, established alliances, status or status quo. How does this affect the kinds of people who are currently in senior posiKons ?

•  AnalyKcs done for the wrong reasons is extremely fragile to economic shock. •  Learning to do analyKcs for the wrong reasons teaches that skill and only that

skill…

analyst first at the chief data officer forum melbourne, august 2015

Data & Analytics