The Consequences ofThe Consequences ofDecentralized Security in a Decentralized Security in a

Cooperative Storage SystemCooperative Storage System

Douglas Thain, Chris Moretti, Paul Madrid,Douglas Thain, Chris Moretti, Paul Madrid,Phil Snowberger, and Jeff HemmesPhil Snowberger, and Jeff Hemmes

University of Notre DameUniversity of Notre Damehttp://www.cse.nd.edu/~cclhttp://www.cse.nd.edu/~ccl

IEEE Workshop on Security in Storage 2005IEEE Workshop on Security in Storage 2005

AbstractAbstract

Suppose that security in storage has been Suppose that security in storage has been deployed at all endpoints.deployed at all endpoints.How does this affect the design of How does this affect the design of distributed storage systems that rely upon distributed storage systems that rely upon these devices?these devices?Clients must become much more:Clients must become much more:– Fault tolerant, adaptive, and self reliant.Fault tolerant, adaptive, and self reliant.– Aware of resource allocation issues.Aware of resource allocation issues.– Helpful to the end user!Helpful to the end user!

Environment: Storage Pool at Notre DameEnvironment: Storage Pool at Notre Dame

Traditional File System SecurityTraditional File System Security

applappl appl

Security Interface

File System Abstraction

disk disk diskfile

Owner ofInode 9842is UID 56

trusted network:PCI

RAIDSAN

MyrinetEthernet

untrusted network:EthernetInternet

placement,replication,reliability

I amJohn Doe!

Decentralized SecurityDecentralized Security

applappl appl

Abstr.

disk disk diskfile

untrusted network:EthernetInternet

Security Security Security

Abstr. Abstr.

placement,replication,reliability

Owner ofFile /foo/baris John Doe

I amJohn Doe!

CooperativeCooperativeStorage Storage SystemSystem

atatNotre DameNotre Dame

What is Cooperative Storage?What is Cooperative Storage?

Many devices bound together that can Many devices bound together that can accomplish more than one device alone.accomplish more than one device alone.– Improve capacity, reliability, performance...Improve capacity, reliability, performance...– Could be one person, or many cooperating users.Could be one person, or many cooperating users.

Key property:Key property:– Each person retains absolute control of their own Each person retains absolute control of their own

resources by setting local policies.resources by setting local policies.– People share and collaborate with others that they People share and collaborate with others that they

know and trust. No free love! No central control!know and trust. No free love! No central control!– However, some resources are set up for the common However, some resources are set up for the common

good by an good by an authorityauthority. (CS workstations usable by . (CS workstations usable by any member of the CS department, says the chair.)any member of the CS department, says the chair.)

file transfer

filesystem

filesystem

filesystem

filesystem

filesystem

filesystem

filesystem

CentralFilesystem

App

Distributed Database Abstraction

Adapter

App

Distributed Filesystem Abstraction

Adapter

App

Cluster administrator controlspolicy on all storage in cluster

UNIX UNIX UNIX UNIX UNIX UNIX UNIX

Workstations owners controlpolicy on each machine.

fileserver

fileserver

fileserver

fileserver

fileserver

fileserver

fileserver

UNIX UNIX UNIX UNIX UNIX UNIX UNIX

???Adapter

3PT

CFS: Central File SystemCFS: Central File System

fileserver

adapteradapter adapter

appl appl appl

file file

file

CFSCFSCFS

ptr ptr

ptr

DSFS: Dist. Shared File SystemDSFS: Dist. Shared File System

fileserver

appl appl

fileserver

fileserver

file file

filefilefile

file filefile

filefile

adapter adapterDSFSDSFS

lookupfile

location

accessdata

DSDB: Dist. Shared DatabaseDSDB: Dist. Shared Database

adapter adapter

appl appl

fileserver

fileserver

file file

filefilefile

file filefile

filefile

databaseserver

file index

query

directaccess

insert

create

file

DSDBDSDB

ApplicationsApplications

Simple and Secure Remote AccessSimple and Secure Remote Access– CDF: Remote Dynamic LinkingCDF: Remote Dynamic Linking– BaBar: Remote Database AccessBaBar: Remote Database Access– LHC: Semantic Remote FilesystemsLHC: Semantic Remote Filesystems

Distributed File SystemsDistributed File Systems– GRAND: Scalable Archive for Online DataGRAND: Scalable Archive for Online Data

Distributed DatabasesDistributed Databases– GEMS: Molecular Dynamics SimulationGEMS: Molecular Dynamics Simulation– CVRL: Biometric Image Storage/AnalysisCVRL: Biometric Image Storage/Analysis

Challenges of DecentralizationChallenges of Decentralization

Unbounded Set of UsersUnbounded Set of Users– There is no global /etc/passwd or /etc/group!There is no global /etc/passwd or /etc/group!

Multiple Identities per UserMultiple Identities per User– Kerberos creds from Notre Dame / Wisconsin.Kerberos creds from Notre Dame / Wisconsin.– GSI creds from ND/UW/DOE/NCSA.GSI creds from ND/UW/DOE/NCSA.

New Decision PointsNew Decision Points– Placement decision made, but action fails!Placement decision made, but action fails!– Directory op succeeds, but file creation fails!Directory op succeeds, but file creation fails!

Unexpected Policy CouplingUnexpected Policy Coupling– Data placement may affect access control!Data placement may affect access control!

Outline of PaperOutline of Paper

Centralized vs Decentralized SecurityCentralized vs Decentralized SecurityArchitecture of Cooperative StorageArchitecture of Cooperative StorageBasic Security MechanismBasic Security Mechanism– Problem: Complexity Confuses!Problem: Complexity Confuses!– Detail: Reservation RightDetail: Reservation Right

ChallengesChallenges– Authorization in Distributed File SystemsAuthorization in Distributed File Systems– Logistics of Third Party TransferLogistics of Third Party Transfer– Mechanisms for Active StorageMechanisms for Active Storage– Semantics of Distributed Group ManagementSemantics of Distributed Group Management

Basic Security MechanismBasic Security Mechanism

Negotiate an Authentication MethodNegotiate an Authentication Method– Client proposes, server agrees/disagrees.Client proposes, server agrees/disagrees.– Default ordering works for most + manual override.Default ordering works for most + manual override.– Different servers/clients may support diff subsets.Different servers/clients may support diff subsets.

Then, Authenticate via Chosen MethodThen, Authenticate via Chosen Method– May involve challenges, cert exchange, etc...May involve challenges, cert exchange, etc...

Yields a Subject Name for the Session:Yields a Subject Name for the Session:– kerberos:dthain@nd.edukerberos:dthain@nd.edu– globus:/O=NotreDame/CN=DouglasThainglobus:/O=NotreDame/CN=DouglasThain– hostname:hedwig.cse.nd.eduhostname:hedwig.cse.nd.edu– unix:dthainunix:dthain

Authorization MechanismAuthorization Mechanism

Unix Access Controls Are Not SufficientUnix Access Controls Are Not Sufficient– Integer UIDs are not sufficient for principals.Integer UIDs are not sufficient for principals.– Nine owner/group/others bits are restrictive.Nine owner/group/others bits are restrictive.– Mapping from subjects to Unix is a mess.Mapping from subjects to Unix is a mess.

Place Variable Length ACLs on dirs:Place Variable Length ACLs on dirs:globus:/O=NotreDame/CN=DThain RWLAXglobus:/O=NotreDame/CN=DThain RWLAX

kerberos:dthain@nd.edu RWLkerberos:dthain@nd.edu RWL

hostname:*.cs.nd.edu RLhostname:*.cs.nd.edu RL

globus:/O=NotreDame/*globus:/O=NotreDame/* RL RL

Problem: Complexity Confuses!Problem: Complexity Confuses!

For beginning users:For beginning users:– Negotiated authentication makes life easy.Negotiated authentication makes life easy.– Everybody can authenticate in some way.Everybody can authenticate in some way.– Most users don’t think about it first.Most users don’t think about it first.

For advanced users:For advanced users:– Negotiation has unexpected effects.Negotiation has unexpected effects.– What happens when credentials expire?What happens when credentials expire?– For long running / large tasks, better to manually For long running / large tasks, better to manually

specify the authentication mode.specify the authentication mode.– AuthN failure is easier to retry than authZ failure!AuthN failure is easier to retry than authZ failure!

Unexpected authentication is hard to debug.Unexpected authentication is hard to debug.– Full detail logging mode reveals auth algorithm.Full detail logging mode reveals auth algorithm.– Always prominently display subject name in all tools!Always prominently display subject name in all tools!

Problem: Shared NamespaceProblem: Shared Namespacefile

server

globus:/O=NotreDame/* RWLAX

a.out

test.c test.dat

cms.exe

Solution: Reservation (V) RightSolution: Reservation (V) Rightfile

server

O=NotreDame/CN=* V(RWLA)

/O=NotreDame/CN=Monk RWLA

mkdir

a.outtest.c

/O=NotreDame/CN=Monk

mkdir

/O=NotreDame/CN=Ted RWLA

a.outtest.c

/O=NotreDame/CN=Tedmkdir only!

Outline of PaperOutline of Paper

Centralized vs Decentralized SecurityCentralized vs Decentralized SecurityArchitecture of Cooperative StorageArchitecture of Cooperative StorageBasic Security MechanismBasic Security Mechanism– Problem: Complexity Confuses!Problem: Complexity Confuses!– Detail: Reservation RightDetail: Reservation Right

ChallengesChallenges– Authorization in Distributed File SystemsAuthorization in Distributed File Systems– Logistics of Third Party TransferLogistics of Third Party Transfer– Mechanisms for Active StorageMechanisms for Active Storage– Semantics of Distributed Group ManagementSemantics of Distributed Group Management

ptr ptr

ptr

DSFS: Dist. Shared File SystemDSFS: Dist. Shared File System

fileserver

appl appl

fileserver

fileserver

file file

filefilefile

file filefile

filefile

adapter adapterDSFSDSFS

lookupfile

location

accessdata

DSFS LogisticsDSFS Logistics

Consider Creating a File:Consider Creating a File:– Fetch list of resources:Fetch list of resources:

online catalog / static list / user selectedonline catalog / static list / user selected

– Make placement decision:Make placement decision:random / fill in order / user selectedrandom / fill in order / user selected

– Create stub file on dir server. (fail?)Create stub file on dir server. (fail?)– Create actual file on data server. (fail?)Create actual file on data server. (fail?)

Note that two access controls are in play:Note that two access controls are in play:– One controls access to the namespace.One controls access to the namespace.– Another controls access to the data storage.Another controls access to the data storage.

DSFS ApplicationsDSFS Applications

Personal Mass StoragePersonal Mass Storage– Expand your local filesystem to include all the disks Expand your local filesystem to include all the disks

available in a cluster / lab / basement.available in a cluster / lab / basement.

Distributed /tmp for Cluster ComputingDistributed /tmp for Cluster Computing– Harness remote cluster for the duration of a job.Harness remote cluster for the duration of a job.

Multi-User Scalable StorageMulti-User Scalable Storage– Department provides directory, but no space.Department provides directory, but no space.

/O=NotreDame/O=CSE/CN=* RWL/O=NotreDame/O=CSE/CN=* RWL

– Participants provide their own data servers.Participants provide their own data servers./O=NotreDame/O=CSE/CN=JohnDoe RWLA/O=NotreDame/O=CSE/CN=JohnDoe RWLA

– Separates Separates provisioningprovisioning from from accessaccess!!

Dealing with FailureDealing with Failure

Failure to place data is very common!Failure to place data is very common!– Unexpected access controls on device.Unexpected access controls on device.– Device is temporarily unavailable. (reboot?)Device is temporarily unavailable. (reboot?)– Device is newly installed or creds expired.Device is newly installed or creds expired.– Owner changed the sharing policy.Owner changed the sharing policy.

Soln: Client Needs to Model the SystemSoln: Client Needs to Model the System– Track successes and failures on each device.Track successes and failures on each device.– Failed devices are not tried again for a time.Failed devices are not tried again for a time.– Of course, cannot avoid a device forever...Of course, cannot avoid a device forever...

Outline of PaperOutline of Paper

Centralized vs Decentralized SecurityCentralized vs Decentralized SecurityArchitecture of Cooperative StorageArchitecture of Cooperative StorageBasic Security MechanismBasic Security Mechanism– Problem: Complexity Confuses!Problem: Complexity Confuses!– Detail: Reservation RightDetail: Reservation Right

ChallengesChallenges– Authorization in Distributed File SystemsAuthorization in Distributed File Systems– Logistics of Third Party TransferLogistics of Third Party Transfer– Mechanisms for Active StorageMechanisms for Active Storage– Semantics of Distributed Group ManagementSemantics of Distributed Group Management

PINS: Processing in StoragePINS: Processing in Storage

Observation:Observation:– Traditional clusters separate CPU and storage into Traditional clusters separate CPU and storage into

two distinct systems/problems.two distinct systems/problems.– Distributed computing is always some direct Distributed computing is always some direct

combination of CPU and I/O needs.combination of CPU and I/O needs.

Idea: PINSIdea: PINS– Cluster HW is already a tighly integrated complex of Cluster HW is already a tighly integrated complex of

CPU and I/O. Make the SW reflect the HW.CPU and I/O. Make the SW reflect the HW.– Key: Always compute in the same place that the data Key: Always compute in the same place that the data

is located. Leave newly created data in place.is located. Leave newly created data in place.

Compute via Passive StorageCompute via Passive Storage

file server file server file server file server

A B C D (X 200)

S1 S2 S3 S4

Compute Y=F(X)where X={A,B,C,D} F Y1 Y2 Y3 Y4

Compute via Active StorageCompute via Active Storage

file server file server file server file server

A B C D (X 200)

S1 S2 S3 S4

Compute Y=F(X)where X={A,B,C,D} F

Y1 Y2 Y3 Y4

FFF

Technique: Identity BoxingTechnique: Identity Boxing

/ directory ACL:hostname:*.cse.nd.edu RWLXglobus:/O=NotreDame/* RWLX

fileserver

sim.exe

storage owner

in.datIdentity Box:/O=NotreDame/CN=Monk

sim.exeout.dat

client

> open x.nd.edu

> put sim.exe

> put in.dat

> exec sim.exe

> get out.dat

Unified SemanticsUnified Semantics

Same Identity for Exec and Data AccessSame Identity for Exec and Data Access– Stage in data as user X.Stage in data as user X.– Program runs as user X, data is protected.Program runs as user X, data is protected.– Access results as user X.Access results as user X.

Same ACLs for Exec and Data AccessSame ACLs for Exec and Data Access– Need the X right to run a program.Need the X right to run a program.– RX rights – given user can run fixed F(X).RX rights – given user can run fixed F(X).– WX rights – given user can stage in any F(X).WX rights – given user can stage in any F(X).

Outline of PaperOutline of Paper

Centralized vs Decentralized SecurityCentralized vs Decentralized SecurityArchitecture of Cooperative StorageArchitecture of Cooperative StorageBasic Security MechanismBasic Security Mechanism– Problem: Complexity Confuses!Problem: Complexity Confuses!– Detail: Reservation RightDetail: Reservation Right

ChallengesChallenges– Authorization in Distributed File SystemsAuthorization in Distributed File Systems– Logistics of Third Party TransferLogistics of Third Party Transfer– Mechanisms for Active StorageMechanisms for Active Storage– Semantics of Distributed Group ManagementSemantics of Distributed Group Management

Fully Decentralized User GroupsFully Decentralized User Groups

Distributed Orgs Have Complex NeedsDistributed Orgs Have Complex Needs– CMS Collaboration: 10s of institutions, 100s of PIs, CMS Collaboration: 10s of institutions, 100s of PIs,

1000s of graduate students staff.1000s of graduate students staff.– There is no centralized database for CMS.There is no centralized database for CMS.– Local managers add/remove members locally.Local managers add/remove members locally.

Want Storage Systems that Allow Reference to Want Storage Systems that Allow Reference to Groups Managed by Others:Groups Managed by Others:– Allow access to all staff involved in CMS.Allow access to all staff involved in CMS.– Allow access to any NSF program manager.Allow access to any NSF program manager.– Allow access to all CS faculty at ND/Purdue.Allow access to all CS faculty at ND/Purdue.

Fully Decentralized ACLsFully Decentralized ACLs

univ.edu

members

univ.edu

members

filesystem

fileserver

file

client

read

Access Control Listgroup:ccl.nd.edu/faculty RWLgroup:serv.nsf.gov/managers RLgroup:ftp.cern.org/members RL

checkACL

ccl.nd.edu serv.nsf.gov

faculty managers

ftp.cern.org

members

group lookups

group lookups

Challenges of ACLsChallenges of ACLsPerformance / Availability / ConsistencyPerformance / Availability / Consistency– Give the group/ACL owner control.Give the group/ACL owner control.– Specify maximum time for stale data.Specify maximum time for stale data.

Implemented, but continuing experience leads to Implemented, but continuing experience leads to reflection on the semantics.reflection on the semantics.Example: What to do under failures?Example: What to do under failures?– Partial answer: servers fail quickly, client retries up to Partial answer: servers fail quickly, client retries up to

a user-controlled limit.a user-controlled limit.– Consider: Group A gives W access, group B gives R.Consider: Group A gives W access, group B gives R.– What happens when group A is unavailable?What happens when group A is unavailable?– Two very different questions:Two very different questions:

What rights does user X have?What rights does user X have?Can user X perform a read? Can user X perform a read?

Outline of PaperOutline of Paper

Centralized vs Decentralized SecurityCentralized vs Decentralized SecurityArchitecture of Cooperative StorageArchitecture of Cooperative StorageBasic Security MechanismBasic Security Mechanism– Problem: Complexity Confuses!Problem: Complexity Confuses!– Detail: Reservation RightDetail: Reservation Right

ChallengesChallenges– Authorization in Distributed File SystemsAuthorization in Distributed File Systems– Logistics of Third Party TransferLogistics of Third Party Transfer– Mechanisms for Active StorageMechanisms for Active Storage– Semantics of Distributed Group ManagementSemantics of Distributed Group Management

Practical LessonsPractical Lessons

In a system with decentralized security...In a system with decentralized security...

Users need debugging tools!Users need debugging tools!– Simple examples: whoami, rwhoamiSimple examples: whoami, rwhoami

Client software must become “heavier”Client software must become “heavier”– Must carefully parse a vast array of errors.Must carefully parse a vast array of errors.– Must maintain a model of remote devices.Must maintain a model of remote devices.

High level names must be used deep within the High level names must be used deep within the system software stack.system software stack.– Run processes with subject name, not Unix UID.Run processes with subject name, not Unix UID.

For more information...For more information...

Cooperative Computing LabCooperative Computing Lab

http://www.cse.nd.edu/~cclhttp://www.cse.nd.edu/~ccl

Cooperative Computing ToolsCooperative Computing Tools

http://http://www.cctools.orgwww.cctools.org

Douglas ThainDouglas Thain– dthain@cse.nd.edudthain@cse.nd.edu– http://http://www.cse.nd.edu/~dthainwww.cse.nd.edu/~dthain