globus online - xsede• integrate with the globus research cloud ecosystem • write programs that...
TRANSCRIPT
![Page 1: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/1.jpg)
globus online
Enabling Your Campus to Simplify Research Data Management with Globus Online Steve Tuecke and Raj Kettimuthu Computation Institute University of Chicago and Argonne National Laboratory
![Page 2: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/2.jpg)
www.globusonline.org
Hands-on Session
The goal of this session is to show you (hands-on) how to take a resource and turn it into a GO endpoint.
Each of you is provided with an amazon EC2 machine for this tutorial.
![Page 3: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/3.jpg)
www.globusonline.org
Log into your host
Your slip of paper has the host informa1on.
Log in as user “xsede12”:
Use the password on the slip of paper.
xsede12 has passwordless sudo privileges.
![Page 4: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/4.jpg)
www.globusonline.org
Setup SSH keys
Upload xsede12's SSH key to Globus Online (/home/xsede12/.ssh/id_rsa.pub).
Each host also has five users: joe, bob, sue, ann, and sam.
Password for joe, bob, sue, ann: the same password as the “xsede12” user.
Password for sam is “xsede12tutorial” for all hosts.
![Page 5: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/5.jpg)
www.globusonline.org
Globus Online Endpoint Setup with Globus Connect Multi-User
XSEDE 2012 July 16, 2012
![Page 6: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/6.jpg)
www.globusonline.org
• What is GCMU? • Multi-user version of Globus Connect • Packages a GridFTP server and MyProxy CA, pre-configured for
use with Globus Online • Why GCMU?
• Create transfer endpoints in minutes • Avoid complex GridFTP install
• To download: https://www.globusonline.org/gcmu/
Globus Connect Multi-User
“We used GCMU to form a campus-wide GSI authentication service spanning multiple servers. Now my users have a fast, easy way to get their data wherever it needs to go, and the setup process was trivial." --University of Michigan
“As a resource admin, I've found GCMU an exceedingly useful tool.... With GCMU, setting up a GridFTP server and handling authentication for multiple users is easy." --Oak Ridge National Lab
![Page 7: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/7.jpg)
www.globusonline.org
GCMU Deployments
7
![Page 8: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/8.jpg)
www.globusonline.org
Globus Connect Multi-User
8
GCMU
![Page 9: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/9.jpg)
www.globusonline.org
Globus Connect Multi User
cd /opt
sudo wget h5p://connect.globusonline.org/linux/stable/globusconnect-‐mul@user-‐latest.tgz
sudo tar xzf globusconnect-‐mul@user-‐latest.tgz
cd gcmu*
sudo ./install
![Page 10: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/10.jpg)
www.globusonline.org
Try doing the following
Create a file called tutorial.txt in /home/joe
Go to the GO Web UI -> Start Transfer
Select endpoint username#xsede12
Activate the endpoint as user “joe” (not xsede12). You should see joe's home directory.(Remember: joe's password is the same as xsede12's)
Transfer to/from the endpoint of the person sitting next to you (activate their endpoint as user “sam”).
Does tutorial.txt show up in /home/sam in the host of the person sitting next to you?
![Page 11: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/11.jpg)
www.globusonline.org
Setup SSH keys
Upload xsede12's SSH key to Globus Online (/home/xsede12/.ssh/id_rsa.pub).
Each host also has five users: joe, bob, sue, ann, and sam.
Password for joe, bob, sue, ann: the same password as the “xsede12” user.
Password for sam is “xsede12tutorial” for all hosts.
![Page 12: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/12.jpg)
www.globusonline.org
GridFTP
• Two channel protocol like FTP • Control Channel
– Command/Response – Used to establish data channels – Basic file system operations eg. mkdir, delete etc
• Data channel – Pathway over which file is transferred – Many different underlying protocols can be used
• MODE command determines the protocol
![Page 13: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/13.jpg)
www.globusonline.org
Third Party Transfer
• Client initiates data transfer between 2 servers
• Client forms control channel with 2 servers.
• Information is routed through the client to establish data channel between the two servers.
• Data flows directly between servers – Client is notified by each server when the
transfer is complete
GridFTP Server B
GridFTP Server A
Client
![Page 14: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/14.jpg)
www.globusonline.org
Daemon / inetd
GridFTP Authentication and Authorization
Client GridFTP Server
Port 2811
ROOT
ROOT
USER
![Page 15: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/15.jpg)
www.globusonline.org
• Installation • Download, untar, configure, make
• Security configuration (server admins) • Obtain and install X.509 host certificate from well-known CA • Configure trust roots
• Security configuration (users) • Obtain and install user certificate from well-known CA • Configure trust roots
• Setup authorization (both users and admins) • DN to local username mapping in gridmap file • '/DC=org/DC=doegrids/OU=People/CN=Rajkumar Kettimuthu
227852' rajk
• Too complex for many users and small labs
Security Configuration
15
![Page 16: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/16.jpg)
www.globusonline.org
• Make GridFTP deployment trivial • GridFTP transfers can be achieved “instantly” even by
non-experts
• Automate the process of configuring security • Avoid the need for any end-user or system administrator
involvement in security configuration
• Reduce burden on both users and administrators
• Eliminate frequent sources of errors in GridFTP configuration and use.
GCMU makes it trivial
16
![Page 17: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/17.jpg)
www.globusonline.org
Globus Connect Multi-User
17
GCMU
![Page 18: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/18.jpg)
www.globusonline.org
• Username is embedded in the certificate, as this cert is used to authenticate with this site only
• “/C=US/O=Globus Consortium/OU=Globus Connect Service/CN=84591482-cba0-11e1-b791-1231381b68a7/CN=rajk”
• Authorization callout extracts username from DN, if the certificate is issued by local CA
User mapping
![Page 19: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/19.jpg)
www.globusonline.org
GridFTP Server A
Data Channel Establishment
GridFTP Server B
Client Connect IP:PORT
![Page 20: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/20.jpg)
www.globusonline.org
Data Channel Authentication
20
GridFTP Server B
GridFTP Server A
Client Cred B
Delegated Cred B
Delegated Cred A
Data Channel
Control Channel
![Page 21: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/21.jpg)
www.globusonline.org
Data Channel Authentication
21
GridFTP Server B
GridFTP Server A
Client Cred B
Delegated Cred B
Delegated Cred A
Has to trust Cred B’s CA
Has to trust Cred A’s CA
Data Channel
Control Channel
![Page 22: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/22.jpg)
www.globusonline.org
Data Channel Authentication
22
GridFTP Server B
GridFTP Server A
Client Cred B
Delegated Cred B
Delegated Cred A
Does not trust Cred B’s CA
Does not trust Cred A’s CA
Data Channel
Control Channel
![Page 23: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/23.jpg)
www.globusonline.org
Data Channel Authentication
23
GridFTP Server B
GridFTP Server A
Client Cred B
Shared secret
Shared secret
Data Channel
Control Channel
Shared secret
![Page 24: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/24.jpg)
www.globusonline.org
Data Channel Security Context (DCSC)
24
GridFTP Server B
GridFTP Server A
Client Cred B
Delegated Cred A
Delegated Cred A
Supports DCSC
Send & accept Cred A for
data channel security
DCSC support not required
DCSC
Data Channel
Control Channel
![Page 25: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/25.jpg)
www.globusonline.org
GO / GCMU Interaction
![Page 26: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/26.jpg)
www.globusonline.org
• Site passwords flow through Globus Online • Globus Online does not store passwords • Just pass along to MyProxy servers at site • Still a security concern for some sites
• OAuth • Sites run a OAuth server • Users enter username and password only on a site’s
webpage • GO gets an X.509 credential via OAuth protocol
OAuth
![Page 27: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/27.jpg)
www.globusonline.org
GCMU with OAuth
cd /opt
• sudo wget h5p://connect.globusonline.org/linux/stable/globusconnect-‐mul@user-‐1.2.0rc2.tgz
• sudo tar xzf globusconnect-‐mul@user-‐1.2.0rc1.tgz
cd gcmu*
Sudo apt-‐get install python-‐flask
sudo ./install
![Page 28: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/28.jpg)
www.globusonline.org
Globus Connect Multi User
MyProxy Online CA
PAM
Local Authen1ca1on System (LDAP, RADIUS, Kerberos etc)
Username password
certifficate
Transfer request
certificate
Step 5
Step 7 Step 8
Step 9
Local Storage
GridFTP Server certificate
Access files
Step 10
Step 11
Authen1ca1on & Data Transfer
Authoriza1on
Step 1 Access Endpoint
GridFTP Server
Campu
s Cluster
GCMU
Globus Online (Hosted Service)
Remote Cluster / User’s PC
Oauth Server
Username password
certificate
certificate
Redirect Step 3
Step 4
Step 6
Username password
Step 2
![Page 29: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/29.jpg)
www.globusonline.org
• GCMU may require firewall configuration • Inbound ports: 2811, 7512, and 50000-51000 • Outbound ports: 50000-51000
• GCMU GridFTP cannot be (easily) used with other GridFTP clients besides GO • GCMU uses a GO-issued cert, not a host cert
• GridFTP has many configuration options • E.g., Gridmap file can be used to allow other certs • Limit access to particular directories
• MyProxy CA default proxy lifetime is 12 hours
Details
![Page 30: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/30.jpg)
www.globusonline.org
• GCMU allows you to serve different sets of users with a single GridFTP server
• For example, some XSEDE resource providers wants to setup a GridFTP server that caters to both XSEDE and non-XSEDE users
• Setup a GCMU on top of XSEDE security infrastructure they already have
• Separate GO endpoints e.g., xsede#ranger associated with xsede myproxy and tacc#ranger associated with local myproxy
• Non-xsede users pick tacc#ranger and enter their tacc credentials to access it
Single GridFTP server for different sets of users
![Page 31: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/31.jpg)
www.globusonline.org
• What should be trivial…
The Challenge: Moving Big Data Easily
31
• … can be painfully tedious and time-consuming
“I need my data over there – at my _____” ( supercomputing
center, campus server, etc.)
Data Source
Data Destination
! Config issues
! Unexpected failure = manual retry
Data Source
Data Destination
“GAAAH!%&@#&”
! Firewall issues
![Page 32: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/32.jpg)
www.globusonline.org
• Reliable file transfer. – Easy “fire-and-forget” transfers – Automatic fault recovery – High performance – Across multiple security domains
• No IT required. – Software as a Service (SaaS)
• No client software installation • New features automatically available
– Consolidated support & troubleshooting – Works with existing GridFTP servers – Globus Connect solves “last mile problem”
• Recommended by XSEDE, Blue Waters, NERSC, ALCF, Advanced Photon Source, many Universities
What is Globus Online?
32
![Page 33: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/33.jpg)
globus online
Globus Online Demo
33
![Page 34: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/34.jpg)
www.globusonline.org
Interactive login to command line interface:
Running commands remotely:
Using CLI with gsissh:
Logging into the CLI
$ ssh [email protected]
$ ssh [email protected] <command>
$ gsissh [email protected] <command>
$ ssh [email protected] scp –r –s 3 -D \ nersc#dtn:~/myfile* mylaptop:~/projects/p1 Task ID: 4a3c471e-edef-11df-aa30-1231350018b1 $ _
![Page 35: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/35.jpg)
www.globusonline.org
Commercial storage service
provider
Na1onal research center
Campus compu1ng center
• Place your data where you want
• Access it from anywhere via different protocols
• Update it, version it, and take snapshots
• Share versions with who you want
• Synchronize among locations
Globus Storage: For when you want to …
Globus Storage volume
Globus Transfer, HTTP/REST, Desktop sync
Alpha
![Page 36: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/36.jpg)
www.globusonline.org
• Integrate with the Globus research cloud ecosystem • Write programs that access/manage:
– user identities, profiles, groups, resources – data, compute and collaboration
… via REST APIs and command line programs
Globus Integrate
Globus Integrate: For when you want to…
Globus Transfer
Globus Storage
Globus Collaborate
Globus Connect Mul@ User
Globus Connect
Globus Nexus Globus Toolkit
Globus Compute
![Page 37: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/37.jpg)
www.globusonline.org
• Visit https://www.globusonline.org/signup to: – Get a free account and start moving files
• Visit www.globusonline.org for: – Tutorials, FAQs, Pro Tips, Troubleshooting – Papers – Case Studies
• Contact [email protected] for: – Help getting started – Help using the service
• Follow us at @globusonline on Twitter and Globus Online on Facebook
For More Information
37
![Page 38: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/38.jpg)
www.globusonline.org
Hands-on Session
The goal of this session is to show you (hands-on) how to take a resource and turn it into a GO endpoint.
Each of you is provided with an amazon EC2 machine for this tutorial.
![Page 39: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/39.jpg)
www.globusonline.org
Log into your host
Your slip of paper has the host informa1on.
Log in as user “xsede12”:
Use the password on the slip of paper.
xsede12 has passwordless sudo privileges.
![Page 40: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/40.jpg)
www.globusonline.org
Setup SSH keys
Upload xsede12's SSH key to Globus Online (/home/xsede12/.ssh/id_rsa.pub).
Each host also has five users: joe, bob, sue, ann, and sam.
Password for joe, bob, sue, ann: the same password as the “xsede12” user.
Password for sam is “xsede12tutorial” for all hosts.
![Page 41: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/41.jpg)
www.globusonline.org
• What is GCMU? • Multi-user version of Globus Connect • Packages a GridFTP server and MyProxy CA, pre-configured for
use with Globus Online • Why GCMU?
• Create transfer endpoints in minutes • Avoid complex GridFTP install
• To download: https://www.globusonline.org/gcmu/
Globus Connect Multi-User
“We used GCMU to form a campus-wide GSI authentication service spanning multiple servers. Now my users have a fast, easy way to get their data wherever it needs to go, and the setup process was trivial." --University of Michigan
“As a resource admin, I've found GCMU an exceedingly useful tool.... With GCMU, setting up a GridFTP server and handling authentication for multiple users is easy." --Oak Ridge National Lab
![Page 42: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/42.jpg)
www.globusonline.org
GCMU Deployments
42
![Page 43: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/43.jpg)
www.globusonline.org
Globus Connect Multi-User
43
GCMU
![Page 44: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/44.jpg)
www.globusonline.org
Globus Connect Multi User
cd /opt
sudo wget h5p://connect.globusonline.org/linux/stable/globusconnect-‐mul@user-‐latest.tgz
sudo tar xzf globusconnect-‐mul@user-‐latest.tgz
cd gcmu*
sudo ./install
![Page 45: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/45.jpg)
www.globusonline.org
Try doing the following
Create a file called tutorial.txt in /home/joe
Go to the GO Web UI -> Start Transfer
Select endpoint username#xsede12
Activate the endpoint as user “joe” (not xsede12). You should see joe's home directory.(Remember: joe's password is the same as xsede12's)
Transfer to/from the endpoint of the person sitting next to you (activate their endpoint as user “sam”).
Does tutorial.txt show up in /home/sam in the host of the person sitting next to you?
![Page 46: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/46.jpg)
www.globusonline.org
Setup SSH keys
Upload xsede12's SSH key to Globus Online (/home/xsede12/.ssh/id_rsa.pub).
Each host also has five users: joe, bob, sue, ann, and sam.
Password for joe, bob, sue, ann: the same password as the “xsede12” user.
Password for sam is “xsede12tutorial” for all hosts.
![Page 47: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/47.jpg)
www.globusonline.org
GridFTP
• Two channel protocol like FTP • Control Channel
– Command/Response – Used to establish data channels – Basic file system operations eg. mkdir, delete etc
• Data channel – Pathway over which file is transferred – Many different underlying protocols can be used
• MODE command determines the protocol
![Page 48: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/48.jpg)
www.globusonline.org
Third Party Transfer
• Client initiates data transfer between 2 servers
• Client forms control channel with 2 servers.
• Information is routed through the client to establish data channel between the two servers.
• Data flows directly between servers – Client is notified by each server when the
transfer is complete
GridFTP Server B
GridFTP Server A
Client
![Page 49: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/49.jpg)
www.globusonline.org
Daemon / inetd
GridFTP Authentication and Authorization
Client GridFTP Server
Port 2811
ROOT
ROOT
USER
![Page 50: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/50.jpg)
www.globusonline.org
• Installation • Download, untar, configure, make
• Security configuration (server admins) • Obtain and install X.509 host certificate from well-known CA • Configure trust roots
• Security configuration (users) • Obtain and install user certificate from well-known CA • Configure trust roots
• Setup authorization (both users and admins) • DN to local username mapping in gridmap file • '/DC=org/DC=doegrids/OU=People/CN=Rajkumar Kettimuthu
227852' rajk
• Too complex for many users and small labs
Security Configuration
50
![Page 51: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/51.jpg)
www.globusonline.org
• Make GridFTP deployment trivial • GridFTP transfers can be achieved “instantly” even by
non-experts
• Automate the process of configuring security • Avoid the need for any end-user or system administrator
involvement in security configuration
• Reduce burden on both users and administrators
• Eliminate frequent sources of errors in GridFTP configuration and use.
GCMU makes it trivial
51
![Page 52: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/52.jpg)
www.globusonline.org
Globus Connect Multi-User
52
GCMU
![Page 53: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/53.jpg)
www.globusonline.org
• Username is embedded in the certificate, as this cert is used to authenticate with this site only
• “/C=US/O=Globus Consortium/OU=Globus Connect Service/CN=84591482-cba0-11e1-b791-1231381b68a7/CN=rajk”
• Authorization callout extracts username from DN, if the certificate is issued by local CA
User mapping
![Page 54: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/54.jpg)
www.globusonline.org
GridFTP Server A
Data Channel Establishment
GridFTP Server B
Client Connect IP:PORT
![Page 55: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/55.jpg)
www.globusonline.org
Data Channel Authentication
55
GridFTP Server B
GridFTP Server A
Client Cred B
Delegated Cred B
Delegated Cred A
Data Channel
Control Channel
![Page 56: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/56.jpg)
www.globusonline.org
Data Channel Authentication
56
GridFTP Server B
GridFTP Server A
Client Cred B
Delegated Cred B
Delegated Cred A
Has to trust Cred B’s CA
Has to trust Cred A’s CA
Data Channel
Control Channel
![Page 57: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/57.jpg)
www.globusonline.org
Data Channel Authentication
57
GridFTP Server B
GridFTP Server A
Client Cred B
Delegated Cred B
Delegated Cred A
Does not trust Cred B’s CA
Does not trust Cred A’s CA
Data Channel
Control Channel
![Page 58: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/58.jpg)
www.globusonline.org
Data Channel Authentication
58
GridFTP Server B
GridFTP Server A
Client Cred B
Shared secret
Shared secret
Data Channel
Control Channel
Shared secret
![Page 59: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/59.jpg)
www.globusonline.org
Data Channel Security Context (DCSC)
59
GridFTP Server B
GridFTP Server A
Client Cred B
Delegated Cred A
Delegated Cred A
Supports DCSC
Send & accept Cred A for
data channel security
DCSC support not required
DCSC
Data Channel
Control Channel
![Page 60: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/60.jpg)
www.globusonline.org
GO / GCMU Interaction
![Page 61: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/61.jpg)
www.globusonline.org
• Site passwords flow through Globus Online • Globus Online does not store passwords • Just pass along to MyProxy servers at site • Still a security concern for some sites
• OAuth • Sites run a OAuth server • Users enter username and password only on a site’s
webpage • GO gets an X.509 credential via OAuth protocol
OAuth
![Page 62: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/62.jpg)
www.globusonline.org
GCMU with OAuth
cd /opt
• sudo wget h5p://connect.globusonline.org/linux/stable/globusconnect-‐mul@user-‐1.2.0rc2.tgz
• sudo tar xzf globusconnect-‐mul@user-‐1.2.0rc1.tgz
cd gcmu*
Sudo apt-‐get install python-‐flask
sudo ./install
![Page 63: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/63.jpg)
www.globusonline.org
Globus Connect Multi User
MyProxy Online CA
PAM
Local Authen1ca1on System (LDAP, RADIUS, Kerberos etc)
Username password
certifficate
Transfer request
certificate
Step 5
Step 7 Step 8
Step 9
Local Storage
GridFTP Server certificate
Access files
Step 10
Step 11
Authen1ca1on & Data Transfer
Authoriza1on
Step 1 Access Endpoint
GridFTP Server
Campu
s Cluster
GCMU
Globus Online (Hosted Service)
Remote Cluster / User’s PC
Oauth Server
Username password
certificate
certificate
Redirect Step 3
Step 4
Step 6
Username password
Step 2
![Page 64: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/64.jpg)
www.globusonline.org
• GCMU may require firewall configuration • Inbound ports: 2811, 7512, and 50000-51000 • Outbound ports: 50000-51000
• GCMU GridFTP cannot be (easily) used with other GridFTP clients besides GO • GCMU uses a GO-issued cert, not a host cert
• GridFTP has many configuration options • E.g., Gridmap file can be used to allow other certs • Limit access to particular directories
• MyProxy CA default proxy lifetime is 12 hours
Details
![Page 65: globus online - XSEDE• Integrate with the Globus research cloud ecosystem • Write programs that access/manage: – user identities, profiles, groups, resources – data, compute](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a83124ca49d09a504d8b8/html5/thumbnails/65.jpg)
www.globusonline.org
• GCMU allows you to serve different sets of users with a single GridFTP server
• For example, some XSEDE resource providers wants to setup a GridFTP server that caters to both XSEDE and non-XSEDE users
• Setup a GCMU on top of XSEDE security infrastructure they already have
• Separate GO endpoints e.g., xsede#ranger associated with xsede myproxy and tacc#ranger associated with local myproxy
• Non-xsede users pick tacc#ranger and enter their tacc credentials to access it
Single GridFTP server for different sets of users