introduction to irods christine staiger surfsara amsterdam, 8 th may 2014 10 th oct 2014
TRANSCRIPT
Introduction to iRODS
Christine StaigerSURFsara
Amsterdam, 8th May 2014
10th Oct 2014
iRODS Concepts
The Basics
① Installation
② Tools for administration
③ Adding data and metadata
Contents
3
iRODS Concepts
4
What is iRODS?
5
What does iRODS provide?• Storage virtualization of different disk and tape
storage systems
• A logical namespace across storage locations
• A rule engine to automate data management according to defined policies
• Various client interfaces
6
iRODS Components
7
iRODS Rule Engine
8
The users view:iRODS Data Collections
Collection0
DataObject
DataObject
DataObjectData
Object A
Collection1
DataObject
DataObjectData
Object X
…/Collection0/…/Collection0/DataObject A-B…/Collection0/Collection1/…/Collection0/Collection1/DataObject X-Z
Logical Name Space (iCAT) independent from physical resources
• Similar to Unix Directory and Files• ichmod to set user/group ACLs• Core.re: msiSetDefaultResc(Resource)• .irodsEnv: irodsDefResource=Resource
9
In the background:iRODS Resources
• (Storage) Resource is a Software or Hardware system that stores data
• 3 Resource classes:
HighLatencyLow
Latency
ArchiveResource
Cache Resource
CompoundResource
POSIX like
IO calls
iRODS Resource Groups
10
LowLatency
Cache Resource
POSIX like
IO calls
Resource Group
LowLatency
Cache Resource
LowLatency
Cache Resource
@Server A @Server B @Server C
11
Hands on The Basics
Demo Data GridsVirtual machines: ssh user@IP
IP adress User Password
145.100.58.144 irods1 irods104
145.100.58.147 irods2 irods204
145.100.58.148 irods3 irods304
145.100.58.150 irods4 irods404
145.100.58.155 irods6 irods604
145.100.58.157 irods7 irods704
145.100.58.158 irods8 irods804
145.100.58.160 irods9 irods904
145.100.58.161 irods10 irods1004
Installation and configuration• Execute ./irodssetup in the folder iRODS
• Export the icommandsexport PATH=$PATH:/home/user/iRODS/clients/icommands/bin
• ~/iRODS/config/irods.config
# Database configuration$DATABASE_TYPE = 'postgres';…$DATABASE_HOST = 'localhost';$DATABASE_PORT = '5432';$DATABASE_ADMIN_PASSWORD = 'irods2';$DATABASE_ADMIN_NAME = ’alice';
# iRODS configurationIRODS_HOME = '/home/alice/iRODS';$IRODS_PORT = '1247’;…$IRODS_ADMIN_PASSWORD = ’alice';$IRODS_ICAT_HOST = '';
.irodsEnv file # iRODS personal configuration file.## This file was automatically created during iRODS installation.# Created Mon Jun 18 13:13:27 2012## iRODS server host name:irodsHost ’surfsara-cloud'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource 'demoResc'# Home directory in iRODS:irodsHome '/alicesZone/home/alice'# Current directory in iRODS:irodsCwd '/alicesZone/home/alice'# Account name:irodsUserName ’alice'# Zone:irodsZone ‘alicesZone'
The .irodsEnvdetermines the zone and the user for the icommands-client.You can only use onesuch file at a time.
iRODS i-commands
Documentation: https://wiki.irods.org/index.php/icommands
(we don’t provide a exhaustive list of i-commands)
16
iRODS control• Execute ./irodsctl <arg> in the folder iRODS
• Restarting, starting and stopping the iRODS and iCAT server with• <arg> = restart, start, stop
• Restarting, starting and stopping only the iRODS server with• <arg> = irestart, istart, istop
• Initialise an account and log in with iinit
The icommands client• icommands are automatically installed when you install the iRODS
server
• Users must install the client software to connect to an iRODS server
• Categories:• Informational• UNIX and FTP like• Metadata• Functional
ihelp• ihelp Lists all commands
• ihelp -h Usage : ihelp [-ah] [icommand] Display i-commands synopsis or a particular i-command help text Options are: -h this help -a print the help text for all the i-commands
Run with no options to display a synopsis of the i-commands
iRODS Version 3.3.1 February 2014 ihelp
ienv• Lists the iRODS environment variables.
• Equivalent to “iinit -l”
• Usage : ienv [-h]
NOTICE: Release Version = rods3.3.1, API Version = dNOTICE: irodsHost=surfsara-cloudNOTICE: irodsPort=1247NOTICE: irodsDefResource=demoRescNOTICE: irodsHome=/alicesZone/home/aliceNOTICE: irodsCwd=/alicesZone/home/aliceNOTICE: irodsUserName=aliceNOTICE: irodsZone=alicesZone
iuserinfo• Show information about your iRODS user account or the entered
user
• Usage: iuserinfo [-vVh] [user]
name: aliceid: 10007type: rodsadminzone: alicesZoneinfo: comment: create time: 01396887567: 2014-04-07.18:19:27modify time: 01396887567: 2014-04-07.18:19:27member of group: alicemember of group: public
ilsresc• ilsresc lists iRODS resources and resource-groups
• Usage: ilsresc [-lvVhA] [Name]
alice@sara-cloud:~$ ilsrescdemoResc
alice@sara-cloud:~$ ilsresc –l demoRescresource name: demoResc resc id: 10010 zone: alicesZonetype: unix file system class: archive location: surfsara-cloud vault: /home/alice/iRODS/Vault …
Some commands• Unix-like:
ils, ipwd, icd, imkdir, irm, icp ichmod, ipasswd, irsync, ichksum, imv
• FTP-like:iinit, iput, iget, iexit
• Test: Create folder in the logical name space ipwd
imkdir NewFolderils NewFolderirm –r NewFolder
User and Data administration• Change to admin mode: iadmin• Quit admin mode: q
• Make/delete a user: mkuser, rmuser, moduser (iadmin) mkuser user1 rodsuser (iadmin) moduser user1 password *****
• Parameters and their values: iadmin lt, iadmin lt user_type
• iadmin mkresc, iadmin rmresc
• Create new resource: mkdir Data (in your home, not in the iRODS environment) iadmin mkresc NewDataResc “unix file system” cache 145.100.57.139 “/home/alice/Data”
Put and replicate data
• Add the file /home/login/put1.txt to resource iput -v put1.txt ils -l or ils -L iput -h
• ichksum put1.txt (checksum of copy in Vault) md5sum put1.txt (checksum of file in /home/login)
• iput -R NewDataResc put2.txt ils -L
• Replicate data: irepl -R NewDataResc put1.txt
• Trim number of copies: itrim -S demoResc -N 1 put1.txt
Access Control Lists• Change permission for a user: ichmod read user1 put1.txt List access: ils -A
• Change permissions for group “public”: Make a directory in the iRODS environment: imkdir temp Make directory readable for “public”: ichmod -r read public temp
• ils -A /bobsZone/home/bob/temp: ACL - bob#bobsZone:own g:public#bobsZone:read object Inheritance - Disabled
iRODS Metadata• Commands: imeta, iquest, idbo
• Add metadata: imeta add -d put1.txt “Date” “08.10.2013” imeta add -d put1.txt “AnotherDate” “04.04.2014” imeta ls -d put1.txt
• Remove metadata: imeta rm -d put1.txt “Date” “08.10.2013”
• Querying the iCAT: iquest "SELECT DATA_NAME,DATA_CHECKSUM WHERE
DATA_RESC_NAME like ’%Resc%’” DATA_NAME = put1.txt DATA_CHECKSUM = c16d7d3488677a3348b12eb82795d28c
Federation between data grids• List all existing zones: ils /• Zone A acknowledges Zone B and vice-cersa
iadmin mkzone B remote Host:Port• Zone A adds remote user from Zone B: iadmin mkuser user#B rodsuser
• Example: Alice and Bob:Alice acknowledges Bob’s zone:
iadmin mkzone bobsZone remote 145.100.57.74:1247 Bob acknowledges Alice’s zone:
iadmin mkzone alicesZone remote 145.100.57.66:1247 Alice creates remote user Bob:
iadmin mkuser bob#bobsZone rodsuser Now Bob can list and alter files in Alice’s zone: ils /alicesZone
Data Replication between grids• Bob copies a file from his zone to his account in Alice’s zone irsync -R demoResc i:/bobsZone/home/bob/put1.txt i:/alicesZone/home/bob#bobsZone/put1.txt -R specifies the target resource
• Bob copies a file from his home directory to Alice’s zone irsync -R demoResc put3.txt i:/alicesZone/home/bob#bobsZone/put3.txt
• Bob copies a file from Alice’s zone to his home directory mkdir test irsync i:/alicesZone/home/bob#bobsZone/put3.txt test/put3.txt
• Bob deletes a file in Alice’s zone irm /alicesZone/home/bob#bobsZone/put3.txt
Thank you !
Appendix
Trouble shooting
ERROR: putUtil: put error for /alicesZone/home/bob#bobsZone/put1.txt, status = -9000 status = -9000 SYS EXCEED CONNECT CNT
Host configuration problem on the server.Solution: Edit iRODS/server/config/irodsHost
localhost you.rIP.XX.XX surfsara-cloudlocalhost 127.0.0.1
./irodsctl restart