data science at atlassian: the transition towards a data-driven organisation
TRANSCRIPT
![Page 1: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/1.jpg)
Data Science at Atlassian The transition towards a data-driven
organisation
Dr Ilias Flaounas Capital Markets CRC, Sydney
13 May 2015
![Page 2: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/2.jpg)
![Page 3: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/3.jpg)
Atlassian - Trivia• Atlassian is an Australian enterprise software company
specialised in products that support project management and collaboration.
• ~12 years old
• Latest valuation at $US3.3 billion in 2014, (x8 since 2010).
• >1.5K employees
• Headquarters in Sydney
• Best place to work in Australia (The Australian Financial Review, 2014)
![Page 4: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/4.jpg)
• ~43,000 enterprise customers
• Tens of thousands more own $10 starter and free community licenses
![Page 5: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/5.jpg)
JIRA
![Page 6: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/6.jpg)
JIRA
![Page 7: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/7.jpg)
![Page 8: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/8.jpg)
Key Requirements:!
• Support from upper management!
• Some minimum critical mass of customers!
• Serious investmentin infrastructure, people,time
![Page 9: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/9.jpg)
Product Growth Team• Team was formed ~2 years ago
• Main mission: growth!
• Discover value in heaps of behavioural data!
• Champion data-driven decision making
![Page 10: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/10.jpg)
Analytics Infrastructure (v.2)
~3-4TB overall (highly compressed)!
800GB per day produced (uncompressed)
![Page 11: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/11.jpg)
![Page 12: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/12.jpg)
We analyse events, i.e., user actionsTimestamp Instance User Event Attributes
1356958862 mySuperCompany John Login1356968863 mySuperCompany Anne Login1356978862 mySuperCompany John dashboard view1356988862 mySuperCompany Anne dashboard view1357884442 anotherCompany John view issue {issueID = 1234}1357956862 mySuperCompany John create issue {issueID = 1000}1357957862 mySuperCompany Anne view issue {issueID = 1000}1357958862 mySuperCompany Anne comment issue {issueID = 1000}1357959862 mySuperCompany John create issue {issueID = 1001}
… … … … …
![Page 13: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/13.jpg)
The Challenges• Build infrastructure!
- break silos!- replace existing “solutions”!- add instrumentation!- improve data quality!- deal with catch-22…
![Page 14: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/14.jpg)
![Page 15: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/15.jpg)
![Page 16: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/16.jpg)
The Challenges
• Educate!- “analytics is counting”!- help people ask the
correct questions
• Build infrastructure!- break silos!- replace existing “solutions”!- add instrumentation!- improve data quality!- deal with catch-22…
![Page 17: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/17.jpg)
JIRA Agile!Which Agile report is used the most?
![Page 18: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/18.jpg)
However, counting is hard…
Report 1 Report 2 Report 3 Report 4 Report 5 Report 6 Report 7 Report 8 Report 9
![Page 19: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/19.jpg)
![Page 20: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/20.jpg)
No usage
![Page 21: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/21.jpg)
No usage Low usage
![Page 22: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/22.jpg)
No usage Low usage Medium usage High usage
![Page 23: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/23.jpg)
… and beyond counting…
![Page 24: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/24.jpg)
![Page 25: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/25.jpg)
Predicting conversionsHow many variables can you measure?
![Page 26: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/26.jpg)
Top customersnone, n/a, home, self, test,
personal, atlassian, “-”, na, private, abc, personal use, freelance,
my company, student, testing, company, no company,
private, mycompany, a, no, test company, noname, testcompany,
self employed, individual, tester, me, “.” , independent, trial, myself, nocompany, xyz
![Page 27: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/27.jpg)
Predicting conversions• Baseline set at 50% • Using only the sign-up information we have an
accuracy of 65%
![Page 28: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/28.jpg)
The Challenges
• Convince- challenge gut feelings- demonstrate value
• Educate!- “analytics is counting”!- help people ask the correct
questions
• Build infrastructure - replace existing “solutions” - data quality - catch-22…
![Page 29: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/29.jpg)
The argument of common sense
How Common Sense Fails Duncan Watts, 2011
κοινὴ αἲσθησις Aristotle, 384–322 BCE
![Page 31: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/31.jpg)
• Source: P. Chopra, The ultimate guide to A/B testing, http://www.smashingmagazine.com, 2010
![Page 32: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/32.jpg)
A/B Tests
• A/B tests allow a constant improvement of products in a measurable and predictive way
• Last year we deployed ~118 experiments.
• The volume of customers, evaluators, visitors, has become a limited and valuable resource
![Page 33: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/33.jpg)
Subject: %%first_name%%, assemble the bitbucketeers!
Subject: Grab the Bitbucketeers...4, 3, 2, 1
LAUNCH!
![Page 34: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/34.jpg)
![Page 35: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/35.jpg)
![Page 36: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/36.jpg)
![Page 37: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/37.jpg)
Do A/B tests slow down development?
“Move fast with stable infra” Mark Zuckerberg
Facebook F8, 2014
“Pauca sed matura” i.e., “Few, but mature” Carl Friedrich Gauss
1777-1855
“Move Fast and Break Things” “Move Fast and Break Things”
![Page 38: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/38.jpg)
Instead of a Conclusion: We simply recycle the scientific method from other fields, e.g., medicine…
Patient User
Medical doctor Product manager
Illness Bad User Experience
Diagnosis Interviews, surveys…
Treatment UX modification
Medical researchers Growth team
Observations Data mining
Medical trials A/B tests
Ethics Privacy Policy
Evidence-basedmedicine
Data-drivendevelopment
![Page 39: Data Science at Atlassian: The transition towards a data-driven organisation](https://reader030.vdocuments.net/reader030/viewer/2022032700/55d31184bb61ebc1308b478d/html5/thumbnails/39.jpg)
Reed Johnson
Ilias Flaounas
Stephen Lee
Shaun Clowes
Graeme Smith
Tim Garnsey
Fardin Sarker
Theo Voronov
Herman Chow
Leo Balan
John Jiang
Rob Sangster
Andrew Wakeling
Martin Smyrk
Kostas Servis
Houda Chehab
Lakshan Perera
We want YOU!