SAS1844 - Securing Hadoop Clusters while Still Retaining Your SanityEvan Kinney, Security Software DeveloperSAS

$whoamiSecurity Software Developer at SASPlatform authn/authz, general securityKerberos; all the time always.Open source developer/contributor/wranglerDinosaur enthusiast

Twitter: @evankinney GitHub: 3vanFreenode: evanosaurus

Why are we here?Hadoop isnt secure by defaultFixing this is non-trivialMistakes made are hard to diagnoseNo consistent story for authentication (yet)Kerberos is hard

Lets talk about the elephant in the room.(this is the first time that pun has ever been used.)When created, Hadoop didnt need securityComponents were almost exclusively single-tenant and isolatedWhat problems do we have?HDFS and MapReduceBlindly trust that a user is who they say they areAllow for arbitrary Java code execution as the JobTracker service accountTrivial to circumvent permissions restrictionsRogue servicesServices dont authenticate each otherDataNodesKnow the block ID? Awesome, here are your dataz!

Okay, greatnow how do we fix it?Kerberos! (surprise)Doesnt require significantly changing the architectureUses symmetric encryptionAllows services to authenticate each otherAllows services to authenticate usersAllows users to authenticate services

Lets talk about the three-headed dog in the room.Kerberos is a standard protocol [RFC 4120] that allows for mutual authentication of entities via the use of a trusted arbiterDeveloped at MIT originally for AFSV4 - 1988[ish]V5 - 1993; 2005Used in a lot more places than most people thinkActive Directory == LDAP + KerberosOnce its working, is usually fairly transparent to users

Some Kerberos Terminologyprincipal: an entity that can authenticate or be authenticated torealm: an administrative partition that contains one or more principals; always given in uppercaseticket: ASN.1 structure containing information about a request as well as authenticators to verify the infoticket granting ticket (TGT): the result of initial authentication with a KDC; used to assert identity in further requestsservice ticket: issued to a principal for use in asserting identity to Kerberized servicescredentials cache (CC): contains one or more tickets for one principalkeytab: holds any number of pre-salted/hashed long-term keys (passwords); generally used for headless/automated serviceskey distribution center (KDC): the trusted arbiter; stores keys for one or more realms and answers authentication requests from principalsticket granting service (TGS): accepts TGT session keys, issues service ticketsauthentication service (AS): accepts salted/hashed password, issues TGT

How does it work?

Hadoop and KerberosHadoop relies on user accounts to enforce ACLs, et al. (if not configured to look things up in LDAP)Generally speaking, a Kerberos principal is not necessarily associated with a POSIX user account (though, in practice, they usually are)Once authenticated via Kerberos, a user is issued a delegation token (specific to Hadoop) for use in further requestsScheduling is hardDefault ticket lifetime is 10 hours, can be renewed for 7 daysMost distributions assume youre running an isolated realm with your own KDC

What about Active Directory?Great for users; not so great for adminsTwo different deployment architectures:only Active Directoryboth user and Hadoop service principals exist as AD objectsCross-realm principals (trusts)Hadoop service principals exist in a pure Kerberos realm, users exist in ADBoth have fun issuesWheres the user data coming from?

Only Active DirectoryTons and tons of objects to create for Hadoop servicesUnless using a vendor-supplied management mechanism, setup is a very manual processUsually requires IT involvement any time changes are madeADs idea of Kerberos != everyone elses

Cross-Realm Principal ArchitectureMuch more complex; harder to debugUnless you configure the KDC with replication (or a backend database that replicates itself), it becomes a massive SPoFYou have to administer the KDCGetting AD to use the correct encryption types is somewhat challengingWindows (i.e. purely SSPI-based) clients tend to not work consistently (if at all)

Speaking of encryption typesThree are the most used today:aes256-cts-hmac-sha1-96aes128-cts-hmac-sha1-96arcfour-hmac-md5Dont use DESor 3DES, preferably but especially not DESCrypto export/import restrictions cause issues with JavaThe unlimited-strength JCE policy files must be present in the JRE to allow the use of aes256-ctsAD wont do AES if the domain functional level (DFL) is lower than Server 2008Almost all other libraries default to using the cipher suites in the order above, unless configured otherwise

Considerations for HadoopKerberos doesnt encrypt your data or trafficCommunication between all DataNodes and NameNode(s) should be isolated and/or protected (via hadoop.rpc.protection)If users have access to the files themselves, ACLs are basically uselessIf they have root/admin access to the serversnone of this matters anywayHadoop services determine what their hostname (and, thus, service principal name) is via reverse DNS (or via fs.default.name, if set)Also, Kerberos itself is very, very dependent upon properly administered DNS records and local client configuration

Considerations for SASSAS/ACCESS Interface to HadoopUses Java, so subject to aforementioned issuesSAS Enterprise Guide, Web-based products (e.g. SAS Visual Analytics), et al.Need to configure sasauth for PAM authenticationNeed to configure PAM to obtain Kerberos credentials on login (via SSSD, pam_krb5, QAS, etc.)If AD: need to configure nsswitch to obtain user info from AD (via SSSD, nss_ldap, etc.)Needed for both SAS and Hadoop

How can this go wrong?Dont try to enumerate them all; sadness will ensueVast majority of issues are eventually attributed to incorrect or missing configurationAdding debug parameters to the JVM invocation will almost always lead you in the right directionsun.security.krb5.debug=truesun.security.jgss.debug=trueHADOOP_JAAS_DEBUG=trueWireshark is invaluable

Common (and/or Particularly Egregious) PitfallsBad principal mapping to local usersIf the user principal attempting to authenticate is from a realm other than the default realm, rules must be set up to indicate that principals from the other realm are to be trusted as being equivalent to local accounts of the same nameUsually only matters if using cross-realm principals (trusts)Consists of a set of regex-like strings used to parse principals into their constituent partsSet in both krb5.conf and Hadoop configskrb5.conf: auth_to_local (defined per-realm)Hadoop: hadoop.security.auth_to_localJava is *supposed* to look in krb5.conf, but it doesnt work

Common (and/or Particularly Egregious) PitfallsUnlimited-strength JCE policy files missing or badAre you sure you put them in the right JRE?Are you sure you put them in all the JREs?Did you download the correct version?Stack traces (with krb5.debug/jgss.debug):javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]

Common (and/or Particularly Egregious) PitfallsClock skew too greatKerberos requires that all parties involved in authentication have their clocks synchronized within 5 minutes of each other (by default)Use chronyd/ntpd against your preferred authoritative time source on the KDC, and have other clients get their time from itIf AD is involved, the PDC is also an NTP server

Common (and/or Particularly Egregious) PitfallsMechanism level: EncryptedData is encrypted using keytype DES3 CBC mode with SHA1-KD but decryption key is of type NULLLong story short: youre using DES; stop it!Actually due to a bug in Java where the RFC wasnt interpreted correctlyhttps://bugs.openjdk.java.net/browse/JDK-8025124Fixed in Java 8 b113 (and current stable)

Questions?