"building data foundations and analytics tools across the product" by crystal widjaja...
TRANSCRIPT
![Page 1: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/1.jpg)
BuildingDataFoundationsandAnalyticsToolsAcrossthe
Product
![Page 2: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/2.jpg)
WhoamI?
● StartedatGO-JEKinJuly2015asthefirst“data”hireFirstday:CreatingaDataDictionarywithoutanyreferencetablesYesterday:Discussions foramoreadvancedexperimentationplatform,
prototyping GrowthROIformulas,QAingnewdatamarts
![Page 3: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/3.jpg)
Agenda
● Infrastructure forScale
● DataModelFoundations
● ToolsforBusinessUsers
![Page 4: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/4.jpg)
Infrastructure
![Page 5: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/5.jpg)
GO-JEKDataToday
~27%*Thisisonlybusinessmetricsdatacollected
byBI
GROWINGDATAVOLUMEPERMONTH
>5000METABASE CARDSANDTABLEAUSHEETS
>450AVGDAILYBUSINESSUSERSON
INTERNALDATA TOOLS
![Page 6: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/6.jpg)
4FULLTIMEDATAWAREHOUSEDEVELOPERS
>30BIDATA ANALYSTS
100sOFMICROSERVICES ACROSSGO-JEK
GO-JEKDataToday
![Page 7: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/7.jpg)
“Thechoicesyoumadeweretherightchoicesgiventhefactsthatyouhadatthetime.”
- Ajey Gore,CTOatGO-JEK
![Page 8: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/8.jpg)
Storage
![Page 9: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/9.jpg)
Storage
![Page 10: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/10.jpg)
crontabs are fun
![Page 11: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/11.jpg)
![Page 12: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/12.jpg)
DataModeling
![Page 13: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/13.jpg)
Moredatatomorepeople
![Page 14: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/14.jpg)
StagingLayer
RAWDataset
IntegrationLayer
Fact/Dimensiondataset
AccessLayer
Summaryandroll-up data
DatamartLayer
Product-specialized dataset
CurrentDataArchitecture
![Page 15: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/15.jpg)
StagingLayer
RAWDataset
IntegrationLayer
Fact/DimensionDataset
AccessLayer
Summaryandroll-up data
DatamartLayer
Product-specialized dataset
CurrentDataArchitecture
Why?1. Transparency2. Standardization
![Page 16: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/16.jpg)
![Page 17: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/17.jpg)
“CanIgetalistofallfull-timedrivers?Iwantto[givethemareward|putthemonabeta
group|interviewthem|…]”
![Page 18: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/18.jpg)
Whatqualitiesmakeadrivera“full-timedriver”?
#ofdaysthedriver logsintotheappinaweek#ofminutesadriverspendsonabooking#ofbookings adriverdoesperdayonavginthepastXweeks#ofminutesadriverspends logged intotheappperday#ofcompletedbookings adriverdoesinaparticularservicemostcommonhour thedriverlogsintotheappinthepastmonth
KeeptheFirstDataLayerFactual
![Page 19: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/19.jpg)
● Star Schema
● Advantages
○ Clean and structured model
MerchantDimension
id nama kategori_merchant
1 WarungBuIis TRADISIONAL
CustomerDimension
id nama nomor_telepon
123 Jo 628112345678
DriverDimension
id nama jenis_kelamin
456 Asep M
457 Doni M
458 Siti F
OrderFact
id id_customer id_driver id_merchant
10001 123 458 1
ItemFact
id id_order nama_item harga
101 10001 NasiGoreng 30000
102 10001 EsTehManis 5000
DriverSearchFact
id id_driver nama status
1 456 Asep Rejected
2 457 Doni Rejected
3 458 Siti Accepted
● Disadvantages
○ Difficulttododatadiscoveryfornon-technicalusers
○ Needs alotofjoins,resultinginhighcomputationalresourceneeds
![Page 20: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/20.jpg)
AppLoginData BidData CompletedBookingData IncomeData DriverProfileData
FactualActivityData
DailyPartitionofDriverActivityand
ProfileDatainDenormalized&NestedForm
TheDataModel
![Page 21: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/21.jpg)
avg_minutes_online_past_3_days total_minutes_online_past_3_days
avg_minutes_online_past_7_days total_days_active_past_3_days
avg_minutes_online_past_30_days total_orders_completed_past_7_days
avg_income_past_3_days total_orders_completed_past_30_days
avg_income_past_7_days total_services_completed_past_7_days
total_completed_ride_past_7_days total_completed_send_past_7_days
foreachdriver_id...
…and+200otherdatapoints
![Page 22: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/22.jpg)
![Page 23: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/23.jpg)
ToolsforScale
![Page 24: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/24.jpg)
![Page 25: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/25.jpg)
LifecycleofaDataPointOneWeekOld
OneMonthOld
3MonthsOld
![Page 26: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/26.jpg)
![Page 27: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/27.jpg)
![Page 28: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/28.jpg)
![Page 29: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/29.jpg)
![Page 30: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/30.jpg)
![Page 31: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/31.jpg)
![Page 32: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/32.jpg)
![Page 33: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/33.jpg)
![Page 34: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/34.jpg)
LetAnalystsDefineEvents
![Page 35: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/35.jpg)
SampleEventstoSaveonCosts
Better samplethatdatapoint...
![Page 36: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/36.jpg)
![Page 37: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/37.jpg)
![Page 38: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/38.jpg)
![Page 39: "Building Data Foundations and Analytics Tools Across The Product" by Crystal Widjaja (GO-JEK)](https://reader030.vdocuments.net/reader030/viewer/2022021422/5a649b167f8b9a31568b595d/html5/thumbnails/39.jpg)
TakeAway● Buildfor theinfrastructureyouhave,notwhatyouthinkyou’llhave
● Buildsimplestep-by-stepdatamodelswithtransparency
● Buildtoolsthatworkforallthedifferent stagesofthecompany