issues securing (big) data

Download Issues Securing (Big) Data

Post on 16-Jul-2015

184 views

Category:

Technology

0 download

Embed Size (px)

TRANSCRIPT

PowerPoint Presentation

Issues Securing Big DataMike Pluta, Sr Technical Architect | April 23, 2015The enclosed materials are highly sensitive, proprietary and confidential. Please use every effort to safeguard the confidentiality of these materials.Please do not copy, distribute, use, share or otherwise provide access to these materials to any person inside or outside DST Systems, Inc. without prior written approval.

This proprietary, confidential presentation is for general informational purposes only and does not constitute an agreement. By making this presentation available to you, we are not granting any express or implied rights or licenses under any intellectual property right.

If we permit your printing, copying or transmitting of content in this presentation, it is under a non-exclusive, non-transferable, limited license, and you must include or refer to the copyright notice contained in this document. You may not create derivative works of this presentation or its content without our prior written permission. Any reference in this presentation to another entity or its products or services is provided for convenience only and does not constitute an offer to sell, or the solicitation of an offer to buy, any products or services offered by such entity, nor does such reference constitute our endorsement, referral, or recommendation.

Our trademarks and service marks and those of third parties used in this presentation are the property of their respective owners.

2015 DST Systems, Inc. All rights reserved.DisclaimerDisclaimerDST has established internal rules around the use of Big DataData flowing into our data lake is partitioned by, what we call, Data DomainsEach DST business unit is in essence at least one Data DomainData Domains serve as the primary method of organizing our permission-ingBig (or not) Data SecurityBy default, one Business Unit is not granted access to anothers dataAgreements between business units are made to access data for purposeInternal Data Scientists are given cross-Business Unit access to dataManagement mandate to secure data which has not been explicitly granted accessWhat This Means4These rules result in a very complex matrix of permissionsExample belowData Doman Business Unit A may be accessed by Business Unit A and Business Unit D. Business Units B and C may not access this Data Domain

Complexity5BU ABU BBU CBU DData DomainBusiness Unit AXXBusiness Unit BXXBusiness Unit CXXXThird Party DataXXLets deal with just text data on a file system in a Linux serverLogical approach is to arrange directories to track with the Data DomainsFor permission-ing, create a group and directory for each Data DomainAssign the group ownership as appropriateSet umask to 007 new files to have u:rw-, g:rw-, o:--- permissionsScenario6sudo useradd buaadmsudo passwd -d buaadm

sudo useradd bubadmsudo passwd -d bubadm

sudo useradd bucadmsudo passwd -d bucadm

sudo useradd budadmsudo passwd -d budadm

sudo useradd tpdadmsudo passwd -d tpdadmDetails Setup Users and Groups7sudo groupadd buagsudo usermod -G buag buaadm

sudo groupadd bubgsudo usermod -G bubg bubadm

sudo groupadd bucgsudo usermod -G bucg bucadm

sudo groupadd budgsudo usermod -G budg budadm

sudo groupadd tpdgsudo usermod -G tpdg tpdadmsudo usermod -a -G buag,bubg,bucg,budg,tpdg dt206031umask 007

cd $HOMEmkdir data

cd datamkdir buamkdir bubmkdir bucmkdir tpd

cd $HOME/data/buatouch bua_file_1touch bua_file_2touch bua_file_3touch bua_file_4touch bua_file_5sudo chown buaadm:buag *Details Setup Files8cd $HOME/data/bubtouch bub_file_1touch bub_file_2touch bub_file_3touch bub_file_4touch bub_file_5sudo chown bubadm:bubg *

cd $HOME/data/buctouch buc_file_1touch buc_file_2touch buc_file_3touch buc_file_4touch buc_file_5sudo chown bucadm:bucg *cd $HOME/data/tpdtouch tpd_file_1touch tpd_file_2touch tpd_file_3touch tpd_file_4touch tpd_file_5sudo chown tpdadm:tpdg *

cd $HOME/datasudo chown buaadm:buag buasudo chown bubadm:bubg bubsudo chown bucadm:bucg bucsudo chown tpdadm:tpdg tpdWhat It Looks Like9

The directory for the Data Domain Business Unit A can be accessed by members of the bua groupHow can we grant additional access to the bud group, but still restrict other groups?Complexity Redux10

BU ABU BBU CBU DData DomainBusiness Unit AXXBusiness Unit BXXBusiness Unit CXXXThird Party DataXXPOSIX Access Control Lists (ACLs) are the answer to our dilemmaNot enabled by default. Needs to be enabled at the filesystem levelmount with the remount and acl options can enablemount o remount o acl /dev/sda5 /homeSee your system administrator for the permanent enableThe Secret Sauce11

setfacl is used to set the ACL for a file or directorygetfacl is used to query and list the ACL of a file or directoryOur specific need:In addition to rwx permissions for the group buag, add rwx permissions for the group budg to the directory buaIn addition to rwx permissions for the group bubg, add rwx permissions for the group budg to the directory bubIn addition to rwx permissions for the group bucg, add rwx permissions for the groups bubg and budg to the directory bucIn addition to rwx permissions for the group tpdg, add rwx permissions for the groups bucg and budg to the directory tpdThe Tools12In addition to rwx permissions for the group buag, add rwx permissions for the group budg to the directory and contents of buasetfacl R --set u::rwx,g::rwx,o::-,g:budg:rwx buaIn addition to rwx permissions for the group bubg, add rwx permissions for the group budg to the directory and contents of bubsetfacl R --set u::rwx,g::rwx,o::-,g:budg:rwx bubIn addition to rwx permissions for the group bucg, add rwx permissions for the groups bubg and budg to the directory and contents of bucsetfacl R --set u::rwx,g::rwx,o::-,g:bubg:rwx,g:budg:rwx bucIn addition to rwx permissions for the group tpdg, add rwx permissions for the groups bucg and budg to the directory and contents of tpdsetfacl R --set u::rwx,g::rwx,o::-,g:bucg:rwx,g:budg:rwx tpdThe Commands13Results14

Hadoop HDFS v2.6 adds POSIX ACLsMake sure to turn it on firsthdfs-site.xml

dfs.namenode.acls.enabledtrue

Reboot the namenodeSet an ACLhdfs dfs -setfacl -m u::rwx,g::rwx,o::-,g:budg:rwx /buaSee the ACLshdfs dfs getfacl /buaHow To Hadoop It15Use a Default ACL for Automatic Application to New Childrensudo setfacl -d --set u::rwx,g::rwx,o::-,g:budg:rwx buasudo setfacl -d --set u::rwx,g::rwx,o::-,g:budg:rwx bubsudo setfacl -d --set u::rwx,g::rwx,o::-,g:bubg:rwx,g:budg:rwx bucsudo setfacl -d --set u::rwx,g::rwx,o::-,g:bucg:rwx,g:budg:rwx tpdAnd in Hadoophadoop fs -setfacl --set d:u::rwx,d:g::rwx,d:o::-,d:g:budg:rwx buahadoop fs -setfacl --set d:u::rwx,d:g::rwx,d:o::-,d:g:budg:rwx bubhadoop fs -setfacl --set d:u::rwx,d:g::rwx,d:o::-,d:g:bubg:rwx,d:g:budg:rwx buchadoop fs -setfacl --set d:u::rwx,d:g::rwx,d:o::-,d:g:bucg:rwx,d:g:budg:rwx tpdOther Goodies16Results With Default ACLs17

Dont forget about the sticky bitMakes it so that only root or the directory owner can delete filessudo chmod +t buaUse the setgid bit to set new files in a directory to have the same group owner as the directory.Very handy when paired with default ACLSsudo chmod g+s buaLast Extra Bits1819