structure data 2014: inverting 80/20: beyond bespoke big data, ari gesher
DESCRIPTION
Presentation from Ari Gesher, Engineering Ambassador, Palantir Technologies #gigaomlive More at http://events.gigaom.com/structuredata-2014/TRANSCRIPT
![Page 1: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/1.jpg)
Wednesday, March 26, 14
![Page 2: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/2.jpg)
INVERTING 80/20: BEYOND BESPOKE BIG DATA
Ari GesherPalantir Technologies
Wednesday, March 26, 14SLIDES BY 3/14 at NOON - TBD.....
![Page 3: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/3.jpg)
INVERTING 80/20Beyond Bespoke Big Data
(or how operating systems change everything)
mailto: [email protected]
twitter: @alephbass
Ari Gesher, Engineering Ambassador
Wednesday, March 26, 14! •! Hiring, hiring, hiring - the only way to have this problem (and it’s a good one to have) is to have a hiring machine that’s kicking ass.! •! Design for growth - make sure your systems (IT, food, etc) can handle exponential growth. Build small, but design scalable! •! Be most worried about long-lead time resources: hiring, leadership, and real estate! •! Culture: how it changes, how to preserve it! •! Discard your old biases - project managers enable scale, but process hinders at small scale
![Page 4: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/4.jpg)
Wednesday, March 26, 14Artillery tables, calculated by people called ‘computers’
![Page 5: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/5.jpg)
Wednesday, March 26, 14Eniac, 1945
![Page 6: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/6.jpg)
Wednesday, March 26, 14
![Page 7: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/7.jpg)
OS/360 becomes THE computer
Wednesday, March 26, 14
![Page 8: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/8.jpg)
Yak Shavers, Inc
BEFORE OPERATING SYSTEMS
Wednesday, March 26, 14
![Page 9: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/9.jpg)
Yak Shavers, Inc
BEFORE OPERATING SYSTEMS
• Programmers had to write the minutiae of hardware, error codes, etc
Wednesday, March 26, 14
![Page 10: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/10.jpg)
Yak Shavers, Inc
BEFORE OPERATING SYSTEMS
• Programmers had to write the minutiae of hardware, error codes, etc
• Writing directly for the hardware of a single computer
Wednesday, March 26, 14
![Page 11: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/11.jpg)
Yak Shavers, Inc
BEFORE OPERATING SYSTEMS
• Programmers had to write the minutiae of hardware, error codes, etc
• Writing directly for the hardware of a single computer
• Programmers started with nothing but primitive functionality
Wednesday, March 26, 14
![Page 12: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/12.jpg)
Yak Shavers, Inc
BEFORE OPERATING SYSTEMS
• Programmers had to write the minutiae of hardware, error codes, etc
• Writing directly for the hardware of a single computer
• Programmers started with nothing but primitive functionality
• Very little of the effort went into building the actual intent of the software
Wednesday, March 26, 14
![Page 13: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/13.jpg)
yak shaving, n
Wednesday, March 26, 14
![Page 14: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/14.jpg)
yak shaving, n1. Any apparently useless activity which, by allowing you to overcome intermediate difficulties, allows you to solve a larger problem.
Wednesday, March 26, 14
![Page 15: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/15.jpg)
yak shavingAny apparently useless activity which, by allowing you to overcome intermediate difficulties, allows
you to solve a larger problem.
Wednesday, March 26, 14
![Page 16: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/16.jpg)
Yaks keep hair
AFTER OPERATING SYSTEMS
Wednesday, March 26, 14
![Page 17: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/17.jpg)
Yaks keep hair
AFTER OPERATING SYSTEMS
• OS API is a clean abstraction, hardware agnostic
Wednesday, March 26, 14
![Page 18: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/18.jpg)
Yaks keep hair
AFTER OPERATING SYSTEMS
• OS API is a clean abstraction, hardware agnostic
• Hardware can change and code still works
Wednesday, March 26, 14
![Page 19: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/19.jpg)
Yaks keep hair
AFTER OPERATING SYSTEMS
• OS API is a clean abstraction, hardware agnostic
• Hardware can change and code still works
• Common functionality (GUI, networking, files) shared by all programs
Wednesday, March 26, 14
![Page 20: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/20.jpg)
Yaks keep hair
AFTER OPERATING SYSTEMS
• OS API is a clean abstraction, hardware agnostic
• Hardware can change and code still works
• Common functionality (GUI, networking, files) shared by all programs
• Majority of effort goes into primary purpose
Wednesday, March 26, 14
![Page 21: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/21.jpg)
THE RIGHT ABSTRACTION
Wednesday, March 26, 14
![Page 22: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/22.jpg)
Wednesday, March 26, 14
![Page 23: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/23.jpg)
Wednesday, March 26, 14
![Page 24: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/24.jpg)
Wednesday, March 26, 14
![Page 25: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/25.jpg)
abstraction creates efficiency
Wednesday, March 26, 14
![Page 26: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/26.jpg)
abstraction creates SAFETY
Wednesday, March 26, 14
![Page 27: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/27.jpg)
abstraction creates innovation
Wednesday, March 26, 14
![Page 28: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/28.jpg)
big data, n1. Data sets that are too large and complex to manipulate or interrogate with standard methods or tools (OED, 2013).
2. A set of technologies and tools designed to address large-scale data analysis problems.
3. A IT buzzword from the early teens of the 21st century.
Wednesday, March 26, 14
![Page 29: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/29.jpg)
Wednesday, March 26, 14
![Page 30: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/30.jpg)
Wednesday, March 26, 14
![Page 31: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/31.jpg)
OPERATING SYSTEM ABSTRACTIONS
Wednesday, March 26, 14
![Page 32: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/32.jpg)
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
Wednesday, March 26, 14
![Page 33: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/33.jpg)
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
• Virtual Machine
Wednesday, March 26, 14
![Page 34: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/34.jpg)
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
• Virtual Machine
• File system
Wednesday, March 26, 14
![Page 35: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/35.jpg)
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
• Virtual Machine
• File system
• Permissions / Access Controls
Wednesday, March 26, 14
![Page 36: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/36.jpg)
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
• Virtual Machine
• File system
• Permissions / Access Controls
• Networking
Wednesday, March 26, 14
![Page 37: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/37.jpg)
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
• Virtual Machine
• File system
• Permissions / Access Controls
• Networking
• GUI Toolkits / UI Frameworks
Wednesday, March 26, 14
![Page 38: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/38.jpg)
Yaks keep hair
BIG DATA ABSTRACTIONS
Wednesday, March 26, 14
![Page 39: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/39.jpg)
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
Wednesday, March 26, 14
![Page 40: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/40.jpg)
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
• Virtual object model / Composition Rules
Wednesday, March 26, 14
![Page 41: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/41.jpg)
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
• Virtual object model / Composition Rules
• Object revisioning
Wednesday, March 26, 14
![Page 42: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/42.jpg)
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
• Virtual object model / Composition Rules
• Object revisioning
• Permissions / Access Control
Wednesday, March 26, 14
![Page 43: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/43.jpg)
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
• Virtual object model / Composition Rules
• Object revisioning
• Permissions / Access Control
• Data Exchange / Translation
Wednesday, March 26, 14
![Page 44: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/44.jpg)
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
• Virtual object model / Composition Rules
• Object revisioning
• Permissions / Access Control
• Data Exchange / Translation
• Interactive, integrated, extensible visualization & analysis environment
Wednesday, March 26, 14
![Page 45: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/45.jpg)
Wednesday, March 26, 14
![Page 46: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/46.jpg)
Wednesday, March 26, 14
![Page 47: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/47.jpg)
Wednesday, March 26, 14
![Page 48: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/48.jpg)
Wednesday, March 26, 14
![Page 49: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/49.jpg)
Wednesday, March 26, 14MOVIE ON NEXT SLIDE - NO AUDIO
![Page 50: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/50.jpg)
Wednesday, March 26, 14MOVIE FILE _ NO AUDIO
![Page 51: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/51.jpg)
Wednesday, March 26, 14MOVIE FILE _ NO AUDIO
![Page 52: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/52.jpg)
FAMILIAR
Wednesday, March 26, 14MOVIE FILE _ NO AUDIO
![Page 53: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/53.jpg)
INTERACTIVE
Wednesday, March 26, 14MOVIE FILE _ NO AUDIO
![Page 54: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/54.jpg)
EXPRESSIVE
Wednesday, March 26, 14MOVIE FILE _ NO AUDIO
![Page 56: Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher](https://reader034.vdocuments.net/reader034/viewer/2022051613/54c7a5e44a7959c0408b4673/html5/thumbnails/56.jpg)
Wednesday, March 26, 14