webfts file transfer web interface for fts3 andrea manzi on behalf of the fts team workshop on cloud...
TRANSCRIPT
![Page 1: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/1.jpg)
WebFTSFile Transfer Web Interface for
FTS3 Andrea Manzi
On behalf of the FTS team
Workshop on Cloud Services for File Synchronisation and Sharing
![Page 2: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/2.jpg)
Overview
The FTS3 service WebFTS features WebFTS cloud integrations
Dropbox CERNBox
Ongoing development Dropping X509 Data management operations
17/11/2014WebFTS: File Transfer Web Interface for FTS3 2
![Page 3: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/3.jpg)
What is WebFTS?
Web based tool to transfer files between grid/cloud storages
Modular protocol support gsiftp, http(s), xrootd and srm Cloud extensions: dropbox, CERNBox
Development funded by
17/11/2014 3WebFTS: File Transfer Web Interface for FTS3
![Page 4: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/4.jpg)
Provide access to leading technology
Based on FTS3FTS3 is the service responsible for distributing the majority of LHC data across the WLCG infrastructureLow level data movement service, responsible for moving sets of files from one site to another while allowing participating sites to control the network resource usageUsed by LHC VOs + many others VOs part of EGI ~20PB monthly transfer volume / ~2.2M files per
day (WLCG) http://dashb-fts-transfers.cern.ch/ui/
17/11/2014WebFTS: File Transfer Web Interface for FTS3 4
![Page 5: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/5.jpg)
WebFTS architecture
WebFTS: File Transfer Web Interface for FTS3
BROWSERBROWSER
REST APIREST API REST API
GFAL2GFAL2
FTS3FTS3
DAVIXDAVIX GSIFTPGSIFTP DROPBOX
DROPBOX ……
WEBFTS.jsWEBFTS.js
17/11/2014 5
![Page 6: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/6.jpg)
Security
Simpler access while keeping the same level of security X509 + Oauth for Dropbox Transparent delegation of credentials
Avoid storing or transferring any sensitive data Open access to all source code All sensitive information is used within the
browser and forgotten
17/11/2014 6WebFTS: File Transfer Web Interface for FTS3
![Page 7: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/7.jpg)
Delegation
Delegation is needed to let WebFTS access the grid on users behalf Users make private key available to
browser Not available via browser API
VOMS extensions acquired by the service on users behalf
Why it’s important Gives the users a service which can access
the grid for them, from a browser, with full VOMS credentials
WebFTS: File Transfer Web Interface for FTS3 17/11/2014 7
![Page 8: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/8.jpg)
Additional Features
Check-summing and file overwriting
Possibility to resubmit transfer jobs or only-failed files transfers.
Storage Endpoints Auto-completion For endpoints published on the BDII (EGI
and WLCG Information System) Support for LFC Registration
File catalog developed at CERN and used by EGI and WLCG
17/11/2014 8WebFTS: File Transfer Web Interface for FTS3
![Page 9: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/9.jpg)
Success Stories
WebFTS has been successfully tested to transfer from/to:
EUDAT B2Stage ( iRODS DSI) Any gsiftp/webdav/xrootd aware grid storage
( DPM, dCache, Castor, EOS, Storm) HPC Titan @ Oak Ridge National Lab
(ongoing) https://www.olcf.ornl.gov/titan/
Under evaluation by LHCb
17/11/2014 9WebFTS: File Transfer Web Interface for FTS3
![Page 10: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/10.jpg)
Landing page and Guided-tour
17/11/2014 10WebFTS: File Transfer Web Interface for FTS3
![Page 11: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/11.jpg)
Credential delegation
17/11/2014 11WebFTS: File Transfer Web Interface for FTS3
![Page 12: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/12.jpg)
Transfer interface
17/11/2014 12WebFTS: File Transfer Web Interface for FTS3
![Page 13: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/13.jpg)
Job status interface
17/11/2014 13WebFTS: File Transfer Web Interface for FTS3
![Page 14: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/14.jpg)
Extension for Dropbox
Nice way import/export data from the grid world Avoid the installation of new software and
uses what the user has already installed Zero development of clients Multiplatform is given for free
Integration with Oauth By delegating to FTS the right to interact
with dropbox on users behalf Achieved using web tech
Which requires the interactivity of a browser
WebFTS: File Transfer Web Interface for FTS3 17/11/2014 14
![Page 15: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/15.jpg)
17/11/2014 15WebFTS: File Transfer Web Interface for FTS3
Extension for Dropbox
![Page 16: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/16.jpg)
Dropbox plugin
Server side the development of a plugin for the metadata management and I/O operations was needed: FTS REST integrates the plugin to perform
metadata management operations FTS3 server uses the plugin to perform
the transfers: GridFTP <-> dropbox Http(s) <-> dropbox
WebFTS: File Transfer Web Interface for FTS3 17/11/2014 16
![Page 17: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/17.jpg)
While Dropbox has been integrated via the implementation of a plugin for CERNBox we waited for the new version with EOS as backend ( CERNBox 2.0) We use EOS access via standard grid
protocols ( e.g. xrootd) We map user credentials to correct EOS
namespace The rest comes for free
CERNBox integration
17/11/2014 17WebFTS: File Transfer Web Interface for FTS3
![Page 18: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/18.jpg)
WebFTS With CERNBox
17/11/2014WebFTS: File Transfer Web Interface for FTS3 18
![Page 19: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/19.jpg)
17/11/2014WebFTS: File Transfer Web Interface for FTS3 19
WebFTS With CERNBox
![Page 20: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/20.jpg)
17/11/2014WebFTS: File Transfer Web Interface for FTS3 20
WebFTS With CERNBox
![Page 21: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/21.jpg)
17/11/2014WebFTS: File Transfer Web Interface for FTS3 21
WebFTS With CERNBox
![Page 22: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/22.jpg)
17/11/2014WebFTS: File Transfer Web Interface for FTS3 22
WebFTS With CERNBox
![Page 23: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/23.jpg)
17/11/2014WebFTS: File Transfer Web Interface for FTS3 23
How can we get rid of the delegation step?
1.An Identity Federation: eduGAIN To allow identity providers to authenticate users
at their own institute (SSO)2.A token translation service : STS
To ask the CA for a certificate for the users3.An “IOTA” Certification Authority
To grant the short lived certificate4.VOMS
To accept the new cert as a VO member
Ongoing developments:Access without X509
![Page 24: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/24.jpg)
EDUGAIN
WebFTS: File Transfer Web Interface for FTS3
Built on existing federations and infrastructures
CERN participates in eduGAIN via SWITCHaai Many NRENs participate in eduGAIN too
17/11/2014 24
![Page 25: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/25.jpg)
Security Token Service (STS)
WebFTS: File Transfer Web Interface for FTS3
• An EMI service• SAML in, X509/VOMS
out17/11/2014 25
![Page 26: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/26.jpg)
“IOTA” CA
WebFTS: File Transfer Web Interface for FTS3 17/11/2014 26
![Page 27: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/27.jpg)
VOMS admin
WebFTS: File Transfer Web Interface for FTS3 17/11/2014 27
![Page 28: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/28.jpg)
Architecture
WebFTSWebFTS
CERN SSOCERN SSOIdPIdP
Cred
entia
lsAtt
ribut
es
Web
Redi
rect
WAY
F SAM
L
VOMSVOMSIdPIdPIdPIdPIdPIdP
GridStorageElement
GridStorageElement
X.509VOMS
STSSTS
IOTACA
IOTACA
SAM
L
X.50
9VO
MS
Slide adapted from Romain Wartel, GDB Sept 2014
28
![Page 29: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/29.jpg)
Pros/Cons
X509-free access to the grid infrastructure With VOMS support Without modifying all the services
Federated single sign on One password to remember Numerous services potentially accessible
But we need Site acceptance
WebFTS: File Transfer Web Interface for FTS3 17/11/2014 29
![Page 30: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/30.jpg)
Not only Transfers..
FTS REST API have been extended to support data management operationsDelete Create/Remove foldersRename
Under integration in WebFTS
17/11/2014 30WebFTS: File Transfer Web Interface for FTS3
Ongoing developments:Data Management
![Page 31: WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing](https://reader034.vdocuments.net/reader034/viewer/2022051316/5697bf7b1a28abf838c838a4/html5/thumbnails/31.jpg)
Online service accessible: https://webfts.cern.ch try now! User certificate in your browser
User guide, F.A.Q: Online guided-tour http://fts3-service.web.cern.ch/
documentation/webfts
Official support & code [email protected] https://github.com/cern-it-sdc-id/webfts
Links
17/11/2014 31WebFTS: File Transfer Web Interface for FTS3