keys for success from streams to queries
TRANSCRIPT
© 2014 MapR Technologies 1© 2014 MapR Technologies
Keys for Success from Streams to QueriesTed Dunning
© 2014 MapR Technologies 2
Contact Information
Ted DunningChief Applications Architect at MapR Technologies
Committer & PMC for Apache’s Drill, Zookeeper & othersVP of Incubator at Apache Foundation
Email [email protected] [email protected]
Twitter @ted_dunning
© 2014 MapR Technologies 3
Goals• Real-time or near-time
– Includes situations with deadlines– Also includes situations where delay is simply undesirable– Even includes situations where delay is just fine
• Micro-services– Streaming is a convenient idiom for design– Micro-services … you know we wanted it– Service isolation is a key requirement
© 2014 MapR Technologies 4
Real-time or Near-time?• The real point is flow versus state (catch me later today)
• One consequence of flow-based computing is real-time and near-time become relatively easy
• Life may be a bitch, but it doesn’t happen in batches!
© 2014 MapR Technologies 6
Agenda• Background / micro-services
• Global requirements
• Scale
© 2014 MapR Technologies 7
A microservice is
loosely coupledwith bounded context
© 2014 MapR Technologies 8
How to Couple Services and Break micro-ness• Shared schemas, relational stores• Ad hoc communication between services• Enterprise service busses• Brittle protocols• Poor protocol versioning• Single PoF schema repositories
Don’t do this!
© 2014 MapR Technologies 9
How to Decouple Services• Use self-describing data • Private database tables• Infrastructural communication between services• Use modern protocols• Adopt future-proof protocol practices
• Use shared storage only where necessary due to scale
© 2014 MapR Technologies 11
What is the Right Structure for Flow Compute?• Traditional message queues?
– Message queues are classic answer– Key feature/bug is out-of-order acknowledgement– Many implementations– You pay a huge performance hit for persistence
• Kafka-esque Logs?– Logs are like queues, but with ordering– Out of order consumption is possible, acknowledgement not so much– Canonical base implementation is Kafka– Performance plus persistence
© 2014 MapR Technologies 12
ScenariosProfile Database
© 2014 MapR Technologies 13
The task
© 2014 MapR Technologies 14
Traditional Solution
© 2014 MapR Technologies 15
What Happens Next?
© 2014 MapR Technologies 16
What Happens Next?
© 2014 MapR Technologies 17
How to Get Service Isolation
© 2014 MapR Technologies 18
New Uses of Data
© 2014 MapR Technologies 19
Scaling Through Isolation
© 2014 MapR Technologies 20
Lessons• De-coupling and isolation are key• Private data stores/tables are important,
– But node local storage of private data is a bug• Propagate events, not table updates
• Note that tables and streams should be as easy as files– It should not be necessary to provision a cluster to get them
© 2014 MapR Technologies 21
ScenariosIoT Data Aggregation
© 2014 MapR Technologies 22
Basic Situation
Each location has many
pumps
Multiple locations
© 2014 MapR Technologies 23
What Does a Pump Look Like
TemperaturePressure
Flow
TemperaturePressureFlow
Winding temperature
VoltageCurrent
© 2014 MapR Technologies 24
Basic Situation
Each location has many
pumps
Multiple locations
© 2014 MapR Technologies 25
Basic Architecture Reflects Business Structure
© 2014 MapR Technologies 26
Lessons• Data architecture should reflect business structure
• Even very modest designs involve multiple data centers
• Schemas cannot be frozen in the real world
• Security must follow data ownership
© 2014 MapR Technologies 27
ScenariosGlobal Data Recovery
© 2014 MapR Technologies 28
© 2014 MapR Technologies 29
© 2014 MapR Technologies 30
© 2014 MapR Technologies 31
© 2014 MapR Technologies 32
Lessons• Streams and tables are primitive building blocks
• Arbitrary number of topics important for simplicity + performance
• Updates happen in many places
• Mobility implies change in replication patterns
• Multi-master updates simplify design massively
© 2014 MapR Technologies 33
Converged Requirements
© 2014 MapR Technologies 34
What Have We Learned?• Need persistence and performance
– Possibly for years and to 100’s of millions t/s• Must have convergence
– Need files, tables AND streams– Need volumes, snapshots, mirrors, permissions and …
• Must have platform security– Cannot depend on perimeter– Must follow business structure
• Must have global scale and scope– Millions of topics for natural designs– Multi-master replication and update
© 2014 MapR Technologies 35
The Importance of Common API’s• Commonality and interoperability are critical
– Compare Hadoop eco-system and the noSQL world• Table stakes
– Persistence– Performance– Polymorphism
• Major trend so far is to adopt Kafka API– 0.9 API and beyond remove major abstraction leaks– Kafka API supported by all major Hadoop vendors
© 2014 MapR Technologies 36
What we do at MapR
© 2014 MapR Technologies 37
Evolution of Data Storage
FunctionalityCompatibility
Scalability
LinuxPOSIX
Over decades of progress,Unix-based systems have set the standard for compatibility and functionality
© 2014 MapR Technologies 38
FunctionalityCompatibility
Scalability
LinuxPOSIX
HadoopHadoop achieves much higher scalability by trading away essentially all of this compatibility
Evolution of Data Storage
© 2014 MapR Technologies 39
Evolution of Data Storage
FunctionalityCompatibility
Scalability
LinuxPOSIX
Hadoop
MapR enhanced Apache Hadoop by restoring the compatibility while increasing scalability and performance
FunctionalityCompatibility
Scalability
POSIX
© 2014 MapR Technologies 40
FunctionalityCompatibility
Scalability
LinuxPOSIX
Hadoop
Evolution of Data Storage
Adding converged tables and streams enhances the functionality of the base file system
© 2014 MapR Technologies 41
http://bit.ly/fastest-big-data
© 2014 MapR Technologies 42
How we do this with MapR• MapR Streams is a C++ reimplementation of Kafka API
– Advantages in predictability, performance, scale– Common security and permissions with entire MapR converged data
platform• Semantic extensions
– A cluster contains volumes, files, tables … and now streams– Streams contain topics– Can have default stream or can name stream by path name
• Core MapR capabilities preserved– Consistent snapshots, mirrors, multi-master replication
© 2014 MapR Technologies 43
MapR core Innovations• Volumes
– Distributed management– Data placement
• Read/write random access file system– Allows distributed meta-data– Improved scaling– Enables NFS access
• Application-level NIC bonding• Transactionally correct snapshots and mirrors
© 2014 MapR Technologies 44
MapR's Containers
Each container contains Directories & files Data blocks
Replicated on servers No need to manage
directly
Files/directories are sharded into blocks, whichare placed into containers on disks
Containers are 16-32 GB segments of disk, placed on nodes
© 2014 MapR Technologies 45
MapR's Containers
Each container has a replication chain
Updates are transactional Failures are handled by
rearranging replication
© 2014 MapR Technologies 46
Container locations and replication
CLDB
N1, N2N3, N2N1, N2N1, N3N3, N2
N1
N2
N3Container location database (CLDB) keeps track of nodes hosting each container and replication chain order
© 2014 MapR Technologies 47
MapR ScalingContainers represent 16 - 32GB of data
Each can hold up to 1 Billion files and directories 100M containers = ~ 2 Exabytes (a very large cluster)
250 bytes DRAM to cache a container 25GB to cache all containers for 2EB cluster
But not necessary, can page to disk Typical large 10PB cluster needs 2GB
Container-reports are 100x - 1000x < HDFS block-reports Serve 100x more data-nodes Increase container size to 64G to serve 4EB cluster
© 2014 MapR Technologies 48
But Wait, There’s More• Directories and files are implemented in terms of B-trees
– Key is offset, value is data blob– Internal transactional semantics guarantees safety and consistency– Layout algorithms give very high layout linearization
• Tables are implemented in terms of B-trees– Twisted B-tree implementation allows virtues of log-structured merge tree
without the compaction delays– Tablet splitting without pausing, integration with file system transactions
• Common security and permissions scheme
© 2014 MapR Technologies 49
And More …• Streams are implemented in terms of B-trees as well
– Topics and consumer offsets are kept in stream, not ZK– Similar splitting technology as MapR DB tables – Consistent permissions, security, data replication
• Standard Kafka 0.9 API• Plans to add OJAI for high-level structuring
• Performance is very high
© 2014 MapR Technologies 50
Example
© 2014 MapR Technologies 51
© 2014 MapR Technologies 52
Lessons• API’s matter more than implementations
• There is plenty of room to innovate ahead of the community
• Posix, HDFS, HBASE all define useful API’s
• Kafka 0.9+ also defines a useful and broadly adopted API
© 2014 MapR Technologies 53
Call to action:
Require convergence
© 2014 MapR Technologies 54
© 2014 MapR Technologies 55
Short Books by Ted Dunning & Ellen Friedman• Published by O’Reilly in 2014 - 2016• For sale from Amazon or O’Reilly• Free e-books currently available courtesy of MapR
http://bit.ly/ebook-real-world-hadoop
http://bit.ly/mapr-tsdb-ebook
http://bit.ly/ebook-anomaly
http://bit.ly/recommendation-ebook
© 2014 MapR Technologies 56
Streaming Architectureby Ted Dunning and Ellen Friedman © 2016 (published by O’Reilly)
Free copies at book signing today
http://bit.ly/mapr-ebook-streams
© 2014 MapR Technologies 57
Thank You!
© 2014 MapR Technologies 58
Q & A@mapr maprtech
Engage with us!
MapR
maprtech
mapr-technologies