test for success: automated testing of sas® metadata ...€¦ · easier to read/write than sas,...

Test for Success: Automated Testing of SAS® Metadata Security ImplementationsPaul HomesMetacoda

About Metacoda

• SAS Alliance Silver Member since 2007

• Provide add-ons to SAS® Software for enhanced metadata visibility and exploitation

• Metacoda Plug-ins (SAS Management Console)

• Custom Tasks (SAS Enterprise Guide & AMO)

• Goals:

• Improve your productivity through enhanced metadata visibility

What is Metadata Security Testing?

… & what can we test?

What is Metadata Security Testing?

Verifying SAS metadata has been secured according to business and I.T. policy requirements

Production

(Lev1)

What can we test?: Users Verify expected users exist:

SAS Administrator,

SAS Trusted User, …

… with expected:

group/role memberships (direct/indirect)

capabilities (indirect)

logins (own/shared)

What can we test?: Groups Verify expected groups exist:

SAS Administrators,

SAS System Services,

SAS General Services, …

… with expected:

group/user members (direct/indirect)

group/role memberships (direct/indirect)

capabilities (indirect)

logins (shared)

What can we test?: Roles Verify expected roles exist:

Metadata Server: Unrestricted,

Enterprise Guide: Advanced,

Visual Analytics: Report Viewing, …

… with expected:

group/user members (direct/indirect)

capabilities (direct/indirect/contributed)

What can we test?: ACTs Verify Access Controls Templates

(ACTs):

Have expected permission patterns:

» Groups / Users

» Permissions

Applied to expected objects

Protected with ACTs and explicit permissions (ACEs)

What can we test?: Applied Access Controls Examine Authorization tabs:

Folders, Servers, ACTs, …

Verify access controls have been applied as expected …

Access Control Templates (ACTs)

Explicit Permissions (ACEs)

What can we test?: Effective Permissions Verify Effective Permissions …

for candidate users / groups

on candidate objects

The “end result” … sensitive to:

Users identity hierarchy (groups)

Objects inheritance path

ACTs & explicit permissions applied to objects in the path

Repository ACT

What can we test?: Golden Rules/Best Practices Best Practice Implementation of SAS® Metadata Security at Customer Sites

in Denmark, Cecily Hoffritz & Johannes Jørgensenhttp://support.sas.com/resources/papers/proceedings11/376-2011.pdf

Very limited use of ACEs [GR#1]

Only groups in ACTs and ACEs (not users) [GR#2]

Only implicit group permission denials (PUBLIC/SASUSERS) [GR#3]

All ACTs are protected

No Group/Role membership/contribution loops

No groups with implicit groups as members (PUBLIC/SASUSERS)

Metadata Security Testing: Why?

… & why re-test regularly?

Metadata Security Testing: Why?A Newly Secured and Tested SAS Platform …

Production

(Lev1)

Metadata Security Testing: Why?Some time later after changes from various user roles …

Production

(Lev1)

… is it still adequately secured?

tomorrow?

next week?

next month?

Metadata Security Testing: Why?How can insecure resources impact you & your organization?

Production

(Lev1)

Reputation ?

Failed regulatory requirements ?

Lost customers ?

$$$ ?

Metadata Security Testing: Why?

Test for consistency across multiple environments …

Production

(Lev1)

Test

(Lev2)Development

(Lev3)

Test

SAS 9.3

(Lev1)

SAS 9.2

(Lev1)

Metadata Security Testing: Why?

Test for consistency during SAS version upgrades …

SAS 9.2

(Lev1)

SAS 9.3

(Lev1)SAS 9.4

(Lev1)

Test Test

Metadata Security Testing Considerations

Metadata Security Testing: Method

How do you perform your testing?

Manually via point & click?

Automatically via code?

How consistent are your manual tests?

Ad-hoc?

Well defined test scripts?

Metadata Security Testing: Coverage

How extensive is your testing?

Handful of sensitive / troublesome objects?

Hundreds / thousands of objects?

What types of things do you test?

Folders, Reports, Stored Procs, Info Maps?

Servers, Logins, Libraries, Tables?

Users, Groups, Roles, Capabilities?

ACT definition & usage, explicit permissions?

Metadata Security Testing: Duration

How long does high coverage testing take?

Weeks?

Days?

Hours?

Seconds?

Metadata Security Testing: Frequency

How often do you perform testing?

Daily?

Weekly?

Monthly?

Annually?

Hardly ever: only when troubleshooting?

Problems with Manual Testing From our experience:

It’s almost exclusively an ad-hoc manual process

It takes too long, it’s inconsistent & it’s error-prone

Consequently it’s not done …

with enough coverage & reliability to detect problems

with enough frequency to detect them promptly

So we looked at how we could automate it ….

Automated Metadata Security Testing

A Metadata Security Testing Framework

An engine that tests metadata against XML Test Specifications

Why XML Test Specifications? Easier to read/write than SAS, Java or .Net code!

Wide variety of plain text or XML editors

Help from XML Schema validation

Can be auto-generated

Checked into Version Control Systems (git, svn, etc.)

Compare differences over time

Test Group/Role Memberships for Users<Users complete=“false"><User required="true" name="sasadm“><DirectGroupMemberships complete="true"><Group required="true" name="SASAdministrators"/>

</DirectGroupMemberships><DirectRoleMemberships complete="true"><Role name="META: Unrestricted Users Role"/>

</DirectRoleMemberships></User><User required="true" name="sastrust">…

</User>…

</Users>

Test Permission Patterns for ACTs<ACTs complete="true"><ACT required="true" repository="Foundation" name="Default ACT"><PermissionPattern complete="true"><Group required="true" repository="Foundation" name="PUBLIC"

permissions="-RM,-WM,-WMM,-CM,-R,-W,-C,-D,-A,-X,-S,-I,-U,-RF,-CT,-DT,-AT"/><Group required="true" name="SASUSERS" permissions="+RM,+WM,+CM"/><Group required="true" name="SASAdministrators"

permissions="+RM,+WM,+CM,+A"/><Group required="true" name="SAS System Services" permissions="+RM,+WM"/>

</PermissionPattern>…</ACT>…

</ACTs>

Test Applied Access Controls for Objects<Objects>

<Object required="true" publicType="ACT" name="Default ACT" ><AccessControls complete="true">

<ACT required="true" name="SAS Administrator Settings"/><Group required="true" name="PUBLIC" permissions="-WM"/>

</AccessControls></Object ><Object required="true" publicType="Folder" parentFolder="/" name="System">

<AccessControls complete="true">…

</AccessControls></Object>

…</Objects>

Test Effective Permissions for Objects<Objects>

<Object required="true" publicType="ACT" name="Default ACT" ><EffectivePermissions>

<Group required="true" name="SASAdministrators" permissions="+RMt,+WMt"/><Group name="SAS System Services" permissions="+RMt,-WMi"/><Group name="SASUSERS" permissions="+RMi,-WMi"/><Group name="PUBLIC" permissions="-RMi,-WMe"/><User name="sasadm" permissions="+RMi,+WMi"/><User name="sasdemo" permissions="+RMi,-WMi"/>

</EffectivePermissions></Object ><Object required="true" publicType="Folder" parentFolder="/" name=“HR">

<EffectivePermissions> … </EffectivePermissions></Object>

…</Objects>

Test Golden Rules / Good Practices

…<AllowOnlyGroupsInACTs/><AllowOnlyGroupsInACEs/><AllowOnlyImplicitGroupDenials/><AllowNoACEs/><AllowNoUnprotectedACTs/><AllowNoGroupMembershipLoops/><AllowNoRoleContributionLoops/><AllowNoGroupsWithImplicitMembers/>…

Testing a Single Environment: Export Once

Today: Export current/desired state asMetadata Security Test XML files

Testing a Single Environment: Test & Repeat

Tomorrow, Next Week, Next Month:Compare current state to desired state using previously exported Metadata Security Test XML files

Consistency Testing Different Environments

Export Metadata Security Test XML files from source environment to test for consistency in target environment.

Summary & Conclusion

Metadata Security Testing at Metacoda Consistency for our software testing environments

Multiple Environments (Multiple SAS versions too!)

Before: inconsistent, infrequent, multiple days of testing

Frequency: every night

Coverage: approx 3,000 tests each

Duration: less than 5 seconds each

Common, cross-environment, cross-version test scripts

Few SAS version / environment specific test scripts

Manual Slow

(hours/days)

Infrequent(every few months)

Inconsistent

Low Coverage

Poor Test Documentation & Audit Logs

v.s. Automated Fast

(seconds/minutes)

Frequent(every day)

Consistent

High Coverage

Integral Test Documentation& Audit Logs

Manual Slow issue detection

(days, weeks, months)

Poor use ofSAS admin time

Every Time We Test!(or not)

v.s. Automated Fast issue detection

(minutes/hours)

Better use ofSAS admin time

Create Initial Tests

Resolve Any Issues

For More Information …

Blog: Testing Conditional Grants in SAS Visual Analyticshttp://platformadmin.com/blogs/paul/2015/09/testing-conditional-grants-sas-va/

Blog: Testing Recommended Practices with SAS Metadata Securityhttp://platformadmin.com/blogs/paul/2015/06/testing-recommended-practices/

Blog: SAS Metadata Security Testinghttp://platformadmin.com/blogs/paul/2014/03/sas-metadata-security-testing/

SAS Global Forum 2014 Paperhttp://support.sas.com/resources/papers/proceedings14/1761-2014.pdf

Questions?

Email: paul.homes@metacoda.com

Blog: http://platformadmin.com/

Twitter: http://www.twitter.com/PaulAtMetacoda

LinkedIn: http://au.linkedin.com/in/paulhomes

Web: http://www.metacoda.com/

Please come & talk to us at the Metacoda stand.