hops - distributed metadata for hadoop
TRANSCRIPT
Hops – Distributed MetaData for Hadoop
Jim Dowling Associate Prof @ KTH
Senior Researcher @ SICSCEO @ Hops AB
BDOOP Meetup, Hadoop Summit, Dublin, 12th April 2016
www.hops.io @hopshadoop
MetaData Services in Hadoop
2
3
Metadata Totem Poles in Hadoop
Eventual Consistency
4
With Many Hadoop Clusters
Cluster 1 Cluster N
MetaDataService
MetaDataService
MetaData Service (Aggregator)
MetaData consistency protocols have O(N) operational complexity.
Case Study: Access Control as a MetaData Service
5
6
Access Control in Relational Databases# Multi-tenancy for alice and bob on db1 and db2
grant all privileges on db1.* to ‘alice'@‘%‘;grant all privileges on db2.* to ‘bob'@‘%‘;
#More fine-grained privilegesgrant SELECT privileges on db2.sensitiveTable to ‘alice'@‘192.168.1.2‘;
Databases ensure the consistency of security and policies using foreign keys.
“drop table db2.sensitiveTable” => delete associated privileges
7
Access Control in Hadoop: Apache Sentry
How do you ensure the consistency of the policies and the data?
[Mujumdar’15]
8
Policy Editor for Sentry
Administrators administer privileges for users
9
Talk Overview• Our Story of Distributed Metadata for Hadoop
• Metadata at work in Hops: Multi-tenancy
• Metadata at work in Hops: HopsWorks
Bill Gates’ biggest product regret?*
Windows Future Storage (WinFS*)
*http://www.zdnet.com/article/bill-gates-biggest-microsoft-product-regret-winfs/
12
HDFS v2
DataNodes
HDFS Client
Journal Nodes Zookeeper
SnapshotNode
ActiveNameNode
StandbyNameNode
Asynchronous Replication of NN LogAgreement on the Active NameNodeFaster Recovery - Cut the NN Log
13
Max Pause times for NameNode Heap Sizes*
Max Pause-Times (ms)
100
1000
10000
10
JVM Heap Size (GB)
25 50 75 100
Unopt
imize
d
Optimized
*OpenJDK or Oracle JVM
14
NameNode and Decreasing Memory Costs
Size (GB)
250
500
1000
Year
2015 2016 2017 2018
Projected Max NameNode JVM Heap Size
2019
0
750
Size of RAM in a COTS $7,000 Rack Server
15
Externalizing the NameNode State• Problem:NameNode not scaling up with lower RAM prices
• Solution:Move the metadata off the JVM Heap
• Move it where?An in-memory storage system that can be efficiently queried and managed. Preferably Open-Source.
• MySQL Cluster (NDB)
16
HopsFS Architecture
NameNodes
NDB
Leader
HDFS Client
HopsFS Client
Load Balancer
DataNodes
17
Pluggable DBs: Data Abstraction Layer (DAL)
NameNode(Apache v2)
DAL API(Apache v2)
NDB-DAL-Impl(GPL v2)
Other DB(Other License)
hops-2.5.0.jar dal-ndb-2.5.0-7.4.7.jar
The Global Lock in the NameNode
18
Apache NameNode InternalsClient: mkdir, getblocklocations, createFile,…..
NameNode
Journal Nodes
Client
Reader1 ReaderN…
Handler1 HandlerM
ConnectionList
Call Queue
Meta Data & In-Memory EditLogFSNameSystem Lock
EditLog Buffer
EditLog1 EditLog2 EditLog3
Listener(Nio Thread)
Responder(Nio Thread)
dfs.namenode.service.handlercount (default 10)
ipc.server.read.threadpool.size (default 1)
…
Handler1 HandlerM… Done RPCs
ackIdsflush
HopsFS NameNode InternalsClient: mkdir, getblocklocations, createFile,…..
NameNode
NDB
Client
Reader1 ReaderN…
Handler1 HandlerM
ConnectionList
Call Queue
inodes block_infos replicas
Listener(Nio Thread)
Responder(Nio Thread)
dfs.namenode.service.handlercount (default 10)
ipc.server.read.threadpool.size (default 1)
…
Handler1 HandlerM…
Done RPCs
ackIds
leases…
DAL-ImplDAL API
HARD PART
21
Consistency: Transactions & Implicit Locking
• Serializabile FS ops using implicit locking of subtrees.
[Hakimzadeh, Peiro, Dowling, ”Scaling HDFS with a Strongly Consistent Relational Model for Metadata”, DAIS 2014]
22
Preventing Deadlock and Starvation
• Acquire FS locks in agreed order using FS Hierarchy. • Block-level operations follow the same agreed order.• No cycles => Freedom from deadlock• Pessimistic Concurrency Control ensures progress
/user/jim/myFilemv
readblock_report
Client DataNodeNameNode
Client
Per Transaction Cache• Reusing the HDFS codebase resulted in too many roundtrips to the database per transaction.
• Cache intermediate transaction results at NameNodes.
24
Sometimes, Transactions Just ain’t Enough• Large Subtree Operations (delete, mv, set-quota) can’t always be executed in a single Transaction.
• 4-phase Protocol• Isolation and Consistency• Aggressive batching• Transparent failure handling• Failed ops retried on new NN.• Lease timeout for failed clients.
Leader Election using NDB• Leader to coordinate replication/lease management• NDB as shared memory for Leader Election of NN.
• No more Zookeeper, yay!25[Niazi, Berthou, Ismail, Dowling, ”Leader Election in a NewSQL Database”, DAIS 2015]
Path Component Caching• Path of length N needs O(N) round-trips to resolve• With our cache, O(1) round-trip for a cache hit
/user/jim/myFile
NDB
getInode(0, “user”) getInode
(1, “jim”) getInode(2, “myFile”)
NameNode
/user/jim/myFile
NDB
validateInodes([(0, “user”), (1,”jim”),(2,”myFile”)])
NameNode
CachegetInodes(“/user/jim/myFile”)
Scalable Blocking Reporting• On 100PB+ clusters, internal maintenance protocol traffic makes up much of the network traffic
• Block Reporting - Leader Load Balances- Work-steal when exiting
safe-mode
SafeBlocks
DataNodes
NameNodes
NDB
Leader
Blocks
work steal
HopsFS Performance
28
29
HopsFS Metadata Scaleout
Assuming 256MB Block Size, 100 GB JVM Heap for Apache Hadoop
30
HopsFS Throughput (Spotify Workload)
Experiments performed on AWS EC2 with enhanced networking and C3.8xLarge instances
Hops-YARN
31
32
YARN Architecture
NodeManagers
YARN Client
Zookeeper Nodes
ResourceMgr StandbyResourceMgr
1. Master-Slave Replication of RM State2. Agreement on the Active ResourceMgr
33
NDB
ResourceManager– Monolithic but Modular
ApplicationMasterService
ResourceTrackerService
Scheduler
ClientService
YARN Client
AdminService
Security
Cluster State
HopsResourceTracker
Cluster State
HopsScheduler
NodeManagerNodeManagerYARN Client App MasterApp Master
ResourceManager
34
Hops-YARN Architecture
ResourceMgrs
NDB
Scheduler
YARN Client
NodeManagers
Resource Trackers Leader Election forFailed Scheduler
What do we do with all this Metadata?
35
Hops MetaData Tree
36
HopsFSHopsYARN
NDB
ProjectsDataSets
Hops Users
ProvenanceSearch
HistoryServiceExt-Metadata
37
Problem: Need Cluster per Sensitive DataSet
NSA DataSet
User DataSet
has access to
has access to
Alice can copy/cross-link between data sets
Alice has only one Kerberos Identity. Dynamic Roles not supported in Hadoop.
Alice
38
Solution: Project-Specific UserIDs
Project NSA
Project UsersMember of
NSA__Alice
Users__Alice
Member of
HDFS enforcesaccess control
39
Sharing DataSets with HopsWorks
Project NSA
Project UsersMember of
DataSetowns
Add members of Project NSA to the DataSet group
NSA__Alice
Users__Alice
Member of
HopsWorks enforces Dynamic Roles
40
NSA__Alice
Authenticate
Users__Alice
HopsWorks
HopsFS
HopsYARN
Projects
SecureImpersonation
41
User• Authentication Provider
- JDBC Realm- 2-Factor Authentication- LDAP
42
Project• Members
- Roles: Owner, Data Scientist
• DataSets - Home project- Can be shared
43
Project Roles• Data Owner Privileges
- Import/Export data- Manage Membership- Share DataSets
• Data Scientist Privileges- Write code- Run code- Request access to DataSets
We delegate administration of privileges to users
45
Sharing DataSets between Projects
The same as Sharing Folders in Dropbox
46
Delegate Access Control to HDFS• HDFS enforces access control- UserID per Project- GroupID per
Project and DataSet
• Metadata Integrity using Foreign Keys- Removing a project removes
all users, groups, and (optionally) DataSets.
47
How ACME Inc. handles Free-Text Search
HDFS
In Theory
Unified Search and Update API
In Practice
Inconsistent Metadata
48
Free Text Search with Consistent Metadata
Free-Text Search
Distributed DatabaseElasticSearch
The Distributed Database is the Single Source of Truth.Foreign keys ensure the integrity of Metadata.
MetaDataDesigner
MetaDataEntry
49
Global Search: Projects and DataSets
50
Project Search: Files, Directories
51
Design your own Extended Metadata
Analytics in HopsWorks
52
53
Batch Job Analytics
Interactive Analytics: Zeppelin
Other Features• Audit Logs
• Erasure Coding Replication
• Online upgrade of Hops (and NDB)
• Automated Installation with Karamel
• Tinker friendly – easy to extend metadata!
55
56
Conclusions• Hops is a next-generation distribution of Hadoop.
• HopsWorks is a frontend to Hops that supports true multi-tenancy, free-text search, interactive analytics with Zeppelin/Flink/Spark, and batch jobs.
• Looking for contributors/committers- Pick-me-ups on GitHub
www.hops.io
The TeamActive: Jim Dowling, Seif Haridi, Tor Björn Minde,
Gautier Berthou, Salman Niazi, Mahmoud Ismail,Kamal Hakimzadeh, Ermias Gebremeskel, Theofilos Kakantousis, Johan Svedlund Nordström, Someya Sayeh, Vasileios Giannokostas, Antonios Kouzoupis, Misganu Dessalegn, Ahmad Al-Shishtawy, Ali Gholami.
Alumni: K. “Sri” Srijeyanthan, Steffen Grohsschmiedt, Alberto Lorente, Andre Moré, Stig Viaene, Hooman Peiro, Evangelos Savvidis, Jude D’Souza, Qi Qi, Gayana Chandrasekara,Nikolaos Stanogias, Daniel Bali, Ioannis Kerkinos,Peter Buechler, Pushparaj Motamari, Hamid Afzali,Wasif Malik, Lalith Suresh, Mariano Valles, Ying Lieu.
Hops
Hops[Hadoop For Humans]
Join us!http://github.com/hopshadoop