enterprise user management · -suppose dev/user wants to use commandline. hive, beeline, ... (tm)...
TRANSCRIPT
Who I am?
2010 Computer Eng. @ I.T.U2010 - DBA2012 - ~ M.Sc. @ BOUN2013 - ~ Blogger @ kamudba.blogspot.com.tr2013 - ~ 11g OCP & Perf. Tuning Cert. Expert2015 - DBA and BigData Administrator
Managing entire bigdata environment in companyAdministration of Exadata databasesOTechMag publication,Talks
2016 –~ Oracle BigData Cert. Impl. Specialist.
Currently enjoying with Big Data
Keep in touch
[email protected] tr.linkedin.com/in/erkanulgeytwitter.com/erkanulgey
Today in short
Establish an environment that lets• People feel working in a familiar environment
Oracle database• People work with standarts
Roles, grants , ..• DBA solve integration problems to company
EUM – What/why?
Security of your business Grouping people Managing access
actually just deny Custom rules
How we do that traditionally?
EUM – What/why?
Sample user management for an Oracle database
Person joins company … wish luck for him
Person added to Active Directory group
Define an OID user maps to Person
Groups premapped to database roles Give access grants to that database role
EUM – Big Data
How about Hadoop?
No OID Not mature Security mechanism Complicated integration of tools Problems of Integration to Company
Where to locate How to connect with other systems Role modeling
Developer/User access methods to Hadoop Define development area Via Commandline, via GUI
How to limit resources to people? – That is an issue for another event
DBA in chaos
EUM – Big Data
Lets start to integrate
GOAL : Design Big Data env. like working in Oracle asap.
First determine the needs. Enable security? ….. YES How users/developers access? How Oracle BDA connects with data sources Role/grant modeling of company What tools that Unix admins use
In next slides, we will see how hadoop admin maintains EUM in a fully secure environment
EUM – Big Data
Lets zoom in detail.
Enable Kerberos Security on Oracle BDA.Prepare user access gates to hadoop
Locate BDA within your engineerd systems.
KerberosEnabledSecurity
Layer
EUM – Big Data
Keep zooming
- Suppose Dev/User wants to use commandline.
Hive, Beeline, Java tools, spark jobs,etc
Hadoop fs
!ACL check!
EUM – Big Data
So far, we see the how a user interacts with hadoop , now lets go further to system level
Establish company role/grant modeling on hadoop side.
In this talk , our environment uses AD Kerberos.
The main step > Users must use a valid Kerberos ticket! to access hadoop.
> Ensure that all cluster and big data components are kerberos enabled on configuration
Assume we have following role modeling on company
Group 1 (members) user1, user2, user3 (access) dbtable1,dbtable2…Group 2 (members) user11, user22, user33 (access) dbtable11,dbtable22…
Remember, grants are given to groups not individual users.
EUM – Big Data
Developer uses shell [EDGENODE]~$setbdaenv -testSetting for BDA Test Environment[EDGENODE]~$beelineJava HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release16/09/11 20:44:57 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it.scan complete in 2msConnecting to jdbc:hive2://HIVEHOST:10000/default;principal=hive/HIVEHOST@HOSTREALM16/09/11 20:44:59 [main]: ERROR transport.TSaslTransport: SASL negotiation failurejavax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]…Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
…Error: Could not open client transport with JDBC Uri: jdbc:hive2://HIVEHOST:10000/default;principal=hive/HIVEHOST@HOSTREALM: GSS initiate failed (state=08S01,code=0)Beeline version 1.1.0-cdh5.7.1 by Apache Hive0: jdbc:hive2://HIVEHOST:10000/default (closed)>
EUM – Big Data
Establish company «role/grant modeling » on hadoop side.
In this talk, user management on BDA done with Centrify.
For the authentication, use Sentry.
Sentry watches HIVE process, applies rules and allow/deny user.
EUM – Big Data
Establish company «role/grant modeling » on hadoop side.
For Hive access,Define hive databasesCreate rolesGive access grants for table/db to rolesMap roles to groups.
0: jdbc:hive2://HIVEHOST:10000/default> show databases;..INFO : OK+------------------------+--+| database_name |+------------------------+--+| db1 || db2 || tmp1 || tmp2 |...+------------------------+--+27 rows selected (0.222 seconds)
0: jdbc:hive2://HIVEHOST:10000/default> show grant role role1;...INFO : OK+------------------------------------------------------+------------+------------+---------+-------------------------+-----------------+------------+---------------+-------------------+----------+--+| database | table | partition | column | principal_name | principal_type | privilege | grant_option | grant_time | grantor |+------------------------------------------------------+------------+------------+---------+-------------------------+-----------------+------------+---------------+-------------------+----------+--+| db1 | | | | role1| ROLE | * | false | 1469108056269000 | -- || hdfs://CLUSTER/HDFSPATH1| | | | role1| ROLE | * | false | 1448268203604000 | -- || db2 | table1 | | | role1| ROLE | select | false | 1466420085615000 | -- || tmp1 | table2 | | | role1| ROLE | select | false | 1466603319236000 | -- || hdfs://CLUSTER/HDFSPATH2| | | | role1| ROLE | * | false | 1471593510770000 | -- |+------------------------------------------------------+------------+------------+---------+-------------------------+-----------------+------------+---------------+-------------------+----------+--+6 rows selected (0.091 seconds)0: jdbc:hive2://HIVEHOST:10000/default>
0: jdbc:hive2://HIVEHOST:10000/default> desc database test;..INFO : OK+----------+----------+-------------------------------------------------+-------------+-------------+-------------+--+| db_name | comment | location | owner_name | owner_type | parameters |+----------+----------+-------------------------------------------------+-------------+-------------+-------------+--+| test | | hdfs://CLUSTER/HDFSPATH/test.db | hive | USER | |+----------+----------+-------------------------------------------------+-------------+-------------+-------------+--+1 row selected (0.078 seconds)0: jdbc:hive2://HIVEHOST:10000/default> show roles;...INFO : OK+-----------------------------+--+| role |+-----------------------------+--+| role_global_dba || role1 || role2 |...+-----------------------------+--+15 rows selected (0.074 seconds)
EUM – Big Data
Establish company «role/grant modeling » on hadoop side.
You can execute grant/revoke statements if you/your group is added to Sentry admin groups
0: jdbc:hive2://HIVEHOST:10000/default> grant select on database tmp1 to role role2;...INFO : OKNo rows affected (0.411 seconds)
EUM – Big Data
Establish company «role/grant modeling » on hadoop side.
You can execute grant/revoke statements if you/your group is added to Sentry admin groups
With Apache Senty, we cannot give grants to users . Only roles are supported.
beeline> grant role admin_role to user erkanul;Error: Error while compiling statement: FAILED: SemanticException Sentry does not allow grant/revoke on: USER (state=42000,code=40000)
beeline> SHOW ROLES;ERROR : Error processing Sentry command: Access denied to erkanul. Server Stacktrace: org.apache.sentry.provider.db.SentryAccessDeniedException: Access denied to erkanul…
EUM – Big Data
Establish company «role/grant modeling » on hadoop side.
The main step is assigning roles to OS groups of user. Here we can map roles to groups.
Important! Sentry must get userid information when assinng role or determine user role whenhive access..
To maintain all users in company to bda , Centrify helps us- Do not define local users on BDA nodes, just make them LISTED- When you issue and ID command it must show like following
[BDANODE]~$id erkanuluid=1146673329(erkanul) gid=1119902816(dba_group) groups=1119902816(dba_group),1146698289(bda_allowed_groups),1119932894(db_bda_allowed_groups_centrify),1119916801(sys_dba_group_centrify)
EUM – Big Data
Establish company «role/grant modeling » on hadoop side.
The main step is assigning roles to OS groups of user. Here we can map roles to groups.
0: jdbc:hive2://HIVEHOST:10000/default> Grant role role1 to group `os_group1`;...INFO : OK
0: jdbc:hive2://HIVEHOST:10000/default> SHOW ROLE GRANT GROUP `os_group1`;...INFO : OK+-----------------------------+---------------+-------------+----------+--+| role | grant_option | grant_time | grantor |+-----------------------------+---------------+-------------+----------+--+| role1 | false | NULL | -- |+-----------------------------+---------------+-------------+----------+--+2 rows selected (0.176 seconds)0: jdbc:hive2://gbbdap35:10000/default>
EUM – Big Data
Developer uses shell [EDGENODE]~$setbdaenv -testSetting for BDA Test Environment[EDGENODE]~$beeline….0: jdbc:hive2://HIVEHOST:10000/default> use tmp1;Error: Error while compiling statement: FAILED: SemanticException No valid privilegesUser dev1 does not have privileges for SWITCHDATABASEThe required privileges: Server=server1->Db=tmp1->Table=*->Column=*->action=insert;Server=server1->Db=tmp1->Table=*->Column=*->action=select; (state=42000,code=40000)0: jdbc:hive2://gbbdap35:10000/default>
EUM – Big Data
Developer uses shell [EDGENODE]~$setbdaenv -testSetting for BDA Test Environment[EDGENODE]~$beeline….0: jdbc:hive2://HIVEHOST:10000/default> use tmp1;...INFO : OKNo rows affected (0.125 seconds)0: jdbc:hive2://HIVEHOST:10000/default> show tables;...INFO : OK+------------------+--+| tab_name |+------------------+--+| table1 || table2 || table3 |..+------------------+--+13 rows selected (0.182 seconds)0: jdbc:hive2://HIVEHOST:10000/default>
EUM – Big Data
Establish company «role/grant modeling » on hadoop side.
With Sentry, you can secure all your hive access.
When a user attempts to create an external table to a location which the user has access on HDFS, hive has also access that location.
Sentry stops «create table» statement.
Bug Alert SENTRY-1001 : Improve usability of URIs and URI privileges
CREATE EXTERNAL TABLE foo LOCATION '/path/to/table‘
(Ensure that Hive has access to that location - ACL)grant all on URI'hdfs://CLUSTER/HDFSPATH' to role role1;
EUM – Big Data
Establish company «role/grant modeling » on hadoop side.
For direct Hdfs access,Define ACLs
Best practise: Define solid paths for hive databases , some devs/apps may want direct access to hive db location
Use hadoop fs –setfacl / getfacl commands and remember subgroups
[EDGENODE]~$setbdaenv -testSetting for BDA Test Environment[EDGENODE]~$ kinit...[EDGENODE]~$hadoop fs -ls /Found 10 itemsdrwxrwxr-x - hdfs supergroup 0 2015-08-21 15:29 /appdrwxrwxr-x - oracle supergroup 0 2016-01-27 10:08 /datadrwxrwxr-x - hdfs supergroup 0 2015-07-28 09:50 /etldrwx------ - hbase hbase 0 2016-08-24 13:03 /hbasedrwxr-xr-x - hdfs supergroup 0 2015-07-08 10:03 /metadatadrwxrwxrwx - solr solr 0 2016-07-15 14:43 /solrdrwxrwxr-x - solr solr 0 2016-08-03 16:36 /solr2drwxr-xr-x - hdfs supergroup 0 2016-05-10 19:24 /systemdrwxrwxrwt - hdfs supergroup 0 2016-09-12 15:20 /tmpdrwxr-xr-x - hdfs supergroup 0 2016-09-01 12:47 /user[EDGENODE]~$
[EDGENODE]~$hadoop fs -getfacl /HDFSPATH/tmp1_dbpath <-- corresponds to hive db path# file: //HDFSPATH/dbpath# owner: hive# group: supergroupuser::rwxgroup::rwxgroup:os_group1:rwxgroup:hive:rwxmask::rwxother::---default:user::rwxdefault:group::rwxdefault:group:os_group1:rwx <-- default setting for newly created filesdefault:group:hive:rwxdefault:mask::rwxdefault:other::---
EUM – Big Data
Establish company «role/grant modeling » on hadoop side.
For direct Hdfs access,Define ACLs
Centrify Alert ! ACL ( and also Sentry) must gather userid information to execute rules
[EDGENODE]~$hadoop fs -ls /HDFSPATH/tmp1_dbpath ls: Permission denied: user=user1, access=EXECUTE, inode="/HDFSPATH/":oracle:supergroup:drwxrwx---
[BDANODE]$id user1id: dev1: No such user
EUM – Big Data
How to entegrate users to HUE. ( we are all familiar with SQL)
Remember ! Never define all users localy on hadoop nodes
Configure HUE with LDAP, enable users to access with their LDAP account, so no need to maintain a HUE password.
In this talk, we want tofilter groups to access BDA and not define them all on HUE. Restrict the menus that they can access.
EUM – Big Data
How to entegrate users to HUE.
Approaches – HUE & LDAP Integration
- Provide LDAP settingsLDAP URL , LDAP Search BaseLDAP Bind User Distinguished NameCheck «Use Search Bind Authentication»
EUM – Big Data
How to entegrate users to HUE.
Approaches – HUE & LDAP Integration
Important properties: LDAP User FilterLDAP Username AttributeLDAP Group FilterLDAP Group Name AttributeLDAP Group Membership Attribute
Lets see some cases for HUE user management
You may need an Active Directory explorer tool
EUM – Big Data
How to entegrate users to HUE.
Approaches – HUE & LDAP Integration
Case 1: Admin can add all users to HUE.
Approach:Just Use a simple LDAP User Filter.
LDAP Username Filter: (objectClass=user)LDAP Username Attribute : sAMAccountName
- ADD the user with AD distinguished name
- Bug alert : HUE-1711 [core] LDAP username import lowercase
EUM – Big Data
How to entegrate users to HUE.
Approaches – HUE & LDAP Integration
Case 2: The groups who access HUE are fixed. But we want to limit the menus they access like security or users page..
Approach:Just Use LDAP User Filter and define groups on HUE and filter menus.
LDAP Username Filter: (&(objectClass=user)(|(memberof=CN=os_group1,OU=Distribution Groups,OU=Groups,OU=AllUsers,LDAPSEARCHBASE)(memberof=CN=os_group2,OU=Distribution Groups,OU=Groups,OU=AllUsers,LDAPSEARCHBASE)(memberof=CN=os_group3,OU=Distribution Groups,OU=Groups,OU=AllUsers,LDAPSEARCHBASE)))LDAP Username Attribute : sAMAccountName
- ADD the group with AD distinguished name
EUM – Big Data
How to entegrate users to HUE.
Approaches – HUE & LDAP Integration
Case 3: The groups who access HUE are incresingBut we want to limit the menus they access like security or users page..
Approach:Define a top AD group and add all the groups
BDA_ALLOWED_GROUPS : group1,group2,…groupN
LDAP Group Filter : (memberOf=CN=BDA_ALLOWED_GROUPS,LDAPSEARCHBASE)LDAP Group Name Attribute : sAMAccountNameLDAP Group Membership Attribute : member
Bug Alert HUE-2040 [useradmin] Synchronize nested ldap groups
[] access INFO 10.242.5.28 erkanul - "POST /useradmin/users/add_ldap_groups HTTP/1.1"[] views DEBUG Importing user CN=OS_Group1,LDAPSEARCHBASE[] views WARNING Could not get LDAP details for users with pattern CN=OS_Group1,LDAPSEARCHBASE
EUM – Big Data
How to entegrate users to HUE.
Approaches – HUE & LDAP Integration
Case 3: The groups who access HUE are incresingBut we want to limit the menus they access like security or users page..
Approach:Define a top AD group and add all the groups
BDA_ALLOWED_GROUPS : group1,group2,…groupN
- For adding user to HUE- Define LDAP User Filter :
(&(objectClass=user)(|(memberof:1.2.840.113556.1.4.1941:=CN=BDA_ALLOWED_GROUPS,LDAPSEARCHBASE)))
LDAP Rule 1.2.840.113556.1.4.1941 -> Recursive add all the users under groups and subgroups
Multiple domains in Company – HUE does not care cross-check domains
EUM – Big Data
How to entegrate users to HUE.
Approaches – HUE & LDAP Integration
For case 1 -> Add user one at a timeFor case 2 -> Refresh/sync LDAP Groups on HUE to get new members
+___________________________________________________________________For those scenarios, new members must be LISTED at Centrify.
For case 3 -> Add new bda user group to BDA_ALLOWED_GROUPSAdd user with * to get all new members from all subgroups
+____________________________________________________________________For this scenario, just add BDA_ALLOWED_GROUPS to Centrify and it will refresh to get new members.Less manuel action
EUM – Big Data
How to entegrate users to HUE.
Approaches – HUE & LDAP Integration
You can change attributes of groups for menu accessBecause we dont want them to see eveything
EUM – Big Data
Lets start to integrate
First determine the needs. Enable security? ….. YES How users/developers access? How Oracle BDA connects with data sources Rol/grant modeling of company What tools that Unix admins use
In next slides, we will see how hadoop admin maintains EUM in a fully secure environment
EUM – Big Data
Yes, We integrated !
- We enabled Kerberos- We designed roles/grant modeling (wrt. Company)- We defined HIVE databases and hadoop file organizations- We gave necessary grants
EUM – Big Data
Note!
- Bug Alert SENTRY-1001 : Improve usability of URIs and URI privileges- Bug Alert HUE-2040 [useradmin] Synchronize nested ldap groups- Bug Alert HUE-1711 [core] LDAP username import lowercase-- Ensure users are LISTED on Centrify ( on BDA nodes )- Define top AD group : BDA_ALLOWED_GROUPS- Be careful about multiple domains
- Create HIVE databases different than /user/hive/warehouse- Ensure all hadoop components use Kerberos Security ( Keytab errors)
BONUS : Establish an EDGENODE and keep developers out of BDA NODES
EUM – Big Data
BONUS : Establish an EDGENODE and keep developers out of BDA NODES
- Create a unix/linux server ( maybe one node from BDA ) - Download client configs from your hadoop cluster- Make configuration settings ( dynamically for prod and test )- Define aliases to command line tools
- Do not use HIVE, it has no security
alias beeline='beeline -u "jdbc:hive2://HIVEHOST:10000/default;principal=hive/HIVEHOST@YOURREALM"'alias hive='echo '\''You can not run hive, disabled for security reasons'\'''alias impala-shell='impala-shell -k -i IMPALAHOST'
settestenv () {export JAVA_HOME=/usr/java/default/export BDAenv="test"export HADOOP_HOME=/usr/lib/hadoopexport HIVE_HOME=/usr/lib/hiveexport HBASE_HOME=/usr/lib/hbaseexport SPARK_HOME=/usr/lib/sparkexport HBASE_CONF_DIR=/home/scripts/oracle/testenv/hbase/hbase-conf/export HIVE_CONF_DIR=/home/scripts/oracle/testenv/hive/hive-conf/export HADOOP_CONF_DIR=/home/scripts/oracle/testenv/hdfs/hadoop-conf/export YARN_CONF_DIR=/home/scripts/oracle/testenv/yarn/yarn-conf/export SPARK_CONF_DIR=/home/scripts/oracle/testenv/spark/spark-confexport HADOOP_CONF_DIR=$HADOOP_CONF_DIR:$HIVE_CONF_DIR:$HBASE_CONF_DIRecho "Setting for BDA Test Environment"bash --init-file /etc/profile}