embracing the monolith
TRANSCRIPT
![Page 1: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/1.jpg)
Doubling down on python to move fast without breaking things.
Embracing the Monolith in Small Teams
Leon Sasson @leonsasson
PyData Chicago 2016
![Page 2: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/2.jpg)
Rise Science
![Page 3: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/3.jpg)
Rise Science
![Page 4: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/4.jpg)
Product Goals
• Sleep Improvement • User Enjoyment
![Page 5: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/5.jpg)
Iterate Fast
Young company, timeline of weeks and days.
![Page 6: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/6.jpg)
Data is core to the product
No data = 😩
![Page 7: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/7.jpg)
Development Cycle
Hypothesis
Exploration
Experiment
Productizing
Evaluate & Analyze
![Page 8: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/8.jpg)
Easy, right?
![Page 9: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/9.jpg)
😓
![Page 10: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/10.jpg)
Obstacles
![Page 11: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/11.jpg)
• Data Silos
![Page 12: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/12.jpg)
• Data Silos • Different Tooling
![Page 13: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/13.jpg)
• Data Silos • Different Tooling • People
![Page 14: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/14.jpg)
• Data Silos • Moving from phase requires different tools • People • "It works on staging"
• Testing data products is hard • Garbage in → Garbage out • Capacity problems
![Page 15: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/15.jpg)
• Data Silos • Moving from phase requires different tools • People • "It works on staging"
Extended Product Cycles
![Page 16: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/16.jpg)
How do we start?
Descriptive, visuals, basic summaries
![Page 17: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/17.jpg)
Step Back
![Page 18: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/18.jpg)
What the organization needs.
Understand problem before getting into solutions
![Page 19: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/19.jpg)
Solution First
Focus is on tech trade-off
![Page 20: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/20.jpg)
Solution First
Focus is on tech trade-off
Problem First
Focus is on making progress for the org
vs.
![Page 21: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/21.jpg)
Solution First
Focus is on tech trade-off
Problem First
Focus is on making progress for the org
vs.
![Page 22: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/22.jpg)
Solution First
Focus is on tech trade-off
Problem First
Focus is on making progress for the org
vs.
![Page 23: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/23.jpg)
Business Optimality
Technical Optimality
![Page 24: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/24.jpg)
![Page 25: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/25.jpg)
What's the least I can do to solve the problem?
![Page 26: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/26.jpg)
What's the least I can do to solve the problem?
You need an architecture compatible with this mindset
![Page 27: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/27.jpg)
Monolithic Architecture
![Page 28: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/28.jpg)
© Martin Fowler: http://martinfowler.com/articles/microservices.html
A monolithic application puts all its functionality into a singles process..
... and scales by replicating the monolith
on multiple servers
![Page 29: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/29.jpg)
© Martin Fowler: http://martinfowler.com/articles/microservices.html
A microservices architecture puts each
element of functionality into a separate service..
... and scales by distributing these services across servers, replicating as
needed
![Page 30: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/30.jpg)
© Martin Fowler: http://martinfowler.com/articles/microservices.html© Martin Fowler: http://martinfowler.com/articles/microservices.html
![Page 31: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/31.jpg)
Django. The Good Things
![Page 32: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/32.jpg)
Reuse Libraries
![Page 33: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/33.jpg)
IPython Notebooks
![Page 34: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/34.jpg)
Reuse your ORM when accessing data.
![Page 35: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/35.jpg)
Pandas, django-pandas
![Page 36: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/36.jpg)
Instrumentation
![Page 37: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/37.jpg)
People
![Page 38: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/38.jpg)
The Problem of Toil
".. manual, repetitive, automatable, tactical, devoid of enduring value, and scales linearly as a service grows.."
![Page 39: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/39.jpg)
Toil-induced negative data culture
![Page 40: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/40.jpg)
Self-Serve Analytics
![Page 41: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/41.jpg)
Breaking Data Silos
![Page 42: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/42.jpg)
Why do Data Silos Happen?
person id date duration
1 2016-08-01 450
2 2016-08-01 426
1 2016-08-02 438
Row
person id date duration
1
2
1
2016-08-01
2016-08-01
2016-08-02
450
426
438
Columnar
![Page 43: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/43.jpg)
Centralizing Data
Segment.com
Backend DB
RedshiftETL
![Page 44: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/44.jpg)
Redshift is fast for aggregations
![Page 45: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/45.jpg)
Out-of-the-box compatible with Postgres
(Mostly..)
![Page 46: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/46.jpg)
Bring data to the people
![Page 47: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/47.jpg)
![Page 48: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/48.jpg)
![Page 49: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/49.jpg)
Positive Feedback Loop on Data Culture
Non tech can access data whenever Data team can focus on bigger problems and act as enablers
![Page 50: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/50.jpg)
Be scrappy.
![Page 51: Embracing the Monolith](https://reader031.vdocuments.net/reader031/viewer/2022022200/58ae20f51a28ab7e4a8b5a87/html5/thumbnails/51.jpg)
Thanks!