catdv stornext archive additions: installation and ... · • the catdv stornext archive additions...

CatDV-StorNext Archive Additions: Installation and Configuration Guide

Quad Logic Systems Ltd

Unit 3 Anglo Office Park

Clarence Road

Speedwell

Bristol BS15 1NT

Tel. +44 (0)117 3012530

Copyright & Trade Mark Acknowledgement Statements

CatDV – SquareBox – http://www.squarebox.co.uk

StorNext – Quantum - http://www.quantum.com

of 26

CatDV-StorNext Archive Additions: Installation and Configuration Guide

1 CatDV-StorNext Archive Solution Overview .......................................................... 3

2 System Requirements ........................................................................................... 3

3 Example Deployment Architecture ........................................................................ 5

3.1 CatDV Pro (for Designated Archive Operators only) OSX and Windows ........ 6

3.2 CatDV Worker (One Macintosh Node Required) ............................................. 6

3.3 CatDV Server OSX or Windows ...................................................................... 6

3.4 StorNext Storage Manager – RHEL5/6 ........................................................... 6

4 Pre-Requisites ....................................................................................................... 7

5 Configuring StorNext ............................................................................................. 8

5.1 SN API Password ............................................................................................ 8

5.2 Policy Folder .................................................................................................... 9

5.3 Storage Policy ................................................................................................ 10

6 Worker Preparation .............................................................................................. 12

6.1 Install Archive Additions ................................................................................. 13

6.2 Create a Worker Folder Share ....................................................................... 13

7 CatDV StorNext Archive Additions: Server Configuration ................................... 13

7.1 License Files .................................................................................................. 13

7.2 Settings .......................................................................................................... 14

7.3 Deploying Worker Actions and Testing Communication ................................ 16

7.4 Additional Configuration Settings ................................................................... 17

7.5 CatDV-StorNext Archive Additions Folder Structure ..................................... 17

8 Install the CatDV Client Additions ........................................................................ 19

9 CatDV StorNext Archive Additions Worker Actions ............................................. 20

9.1 CatDV-StorNext Archive ................................................................................ 20

9.2 CatDV-StorNext Retrieve ............................................................................... 22

10 Troubleshooting ................................................................................................. 25

of 26

1 CatDV-‐StorNext Archive Solution Overview Quantum StorNext provides a SAN File System delivering high-performance file creation, storage and sharing of data. The addition of Storage Manager offers an intelligent means of data life cycle management. The CatDV StorNext Archive Additions solution leverages the power of CatDV Enterprise Server and StorNext Storage Manager to provide an intuitive, archival user interface.

CatDV StorNext Archive Additions manages media location (whether online or committed to tape for long-term storage). Typically a small percentage of archived files will be needed for future events but protection of all assets makes sure that media can be made available when needed. Considering the cost of high bandwidth storage, the archive solution may also be employed for ‘edit with proxy’ workflows where the high definition media is required for final conform tasks.

CatDV provides first class media asset management and is the essential control component for your metadata, including file location and historical events relating to archive operations. CatDV StorNext Archive Additions moves media to and from the StorNext archive when requested. CatDV Server receives tracking metadata detailing the media location at any point in time. Please ensure you have a means of periodic back up of your CatDV Server database, as the information contained in iti is critical for management of the StoNext archive.

The user interface for CatDV-SN Archive consists of three actions to empower nominated operators responsible for archive tasks. The product aims to be simple in use but to incorporate routines for validity checking and reporting to keep the user aware of archive related events.

• The CatDV StorNext Archive Additions comprises several additional programs over a standard CatDV Enterprise Server deployment:

• CatDV clients are used to control archive functions • the CatDV Worker is used for the queuing, querying and StorNext

communication. The user requests are queued for operation on the CatDV Worker node. Some initial checks during job submission ensure the receipt of valid operations e.g. media location and duplicate requests, which will provide early feedback to the operator.

• CatDV StorNext Archive Additions itself is a set of programs to handle data relocation and StorNext communication.

• StorNext Storage Manager is also, clearly a critical component

Each component has its own installer

2 System Requirements The implementation has been tested and is supported with the following component versions

of 26

StorNext Storage Manager

Red Hat Enterprise Server 5/6 (Linux is required for Storage Manager)

StorNext 4.2.1 and above

CatDV-SN Worker Additions

System requirements are as CatDV Worker Mac recommendations:

• OSX 10.6 and above • CatDV Worker 4.x or Worker 5.x

CatDV-SN Archivist Additions CatDV 9 v9 or later (desktop versions: CatDV Standard, CatDV Pro, CatDV Enterprise, CatDV Pegasus)

System requirements are as CatDV Pro recommendations.

CatDV Server At least CatDV Server 6.6.

System requirements are as CatDV Server recommendations.

of 26

3 Example Deployment Architecture

Figure 1 CatDV_SN Archive Software Distribution

of 26

3.1 CatDV Pro (for Designated Archive Operators only) OSX and Windows The user application, CatDV Pro, receives additional tools for archive operations. This only applies to staff designated as archive operators; there are no modifications to existing CatDV Pro users unless this functionality is required.

Archive operators are provided with extra menu options to initiate archive or retrieval requests and to access historic performance data.

Installation of CatDV-SN Archivist Additions is required on Archivists workstations.

3.2 CatDV Worker (One Macintosh Node Required) The CatDV Worker is employed to detect specific attribute changes and launch additional programs related to the task defined. If you have deployed multiple Worker nodes then only one is required to receive the additional archive components and scripts.

Installation of CatDV-SN Worker Additions is required on this machine.

3.3 CatDV Server OSX or Windows There are no additions to CatDV server in the CatDV-SN Archive Solution unless the server is a Macintosh and you wish to run the Worker archiving actions on this machine.

3.4 StorNext Storage Manager – RHEL5/6 Please see StorNext Storage Manager documentation for OS requirements.

There are no additional programs for installation to the StorNext Server. The administrator will need to configure a Storage Manager policy that provides the relocation of media and truncation to comply with the desired workflow. Example configurations will be presented later in the documentation.

of 26

4 Pre-‐Requisites This section describes the items that are required before an installation should be attempted.

1. A CatDV server (v6.6 or above) and clients (v9 or above) installed (on either Mac or Windows)

2. A CatDV worker (v4 or above) installed and running on a Macintosh OSX 10.6 or above

3. CatDV running on appropriate hardware according to CatDV System requirements.

4. Appropriate CatDV path mappings if network paths on the CatDV clients (Pro, Enterprise or Pegasus), Server and Worker are different.

5. StorNext Storage Manager is already functioning and accessible on the network.

6. Appropriate networking so that CatDV clients, the CatDV Server, the CatDV Worker and StorNext can see the footage on the chosen storage.

The CatDV Worker and Server may be on separate machines or potentially on the same machine if it’s a Mac depending on the CatDV deployment. The Worker requires access to the project media in order to manipulate files to and from StorNext. This may consist of fiber or Ethernet accessible storage with SAN agents providing connectivity.

CatDV StorNext Archive Additions communicates using addresses to make installation and configuration simpler. It is therefore important to maintain a static addressing scheme (which is normally the case in the SAN environment). The CatDV StorNext Archive Additions configuration program runs on the nominated CatDV Worker node machine and must be aware of these IP address targets, particularly the metadata controller. User accounts and related passwords will also be required to permit component communication.

It may be useful to perform the following steps to verify the set up above has completed successfully:

• Test storage connectivity to ensure the volumes holding high-resolution media are accessible by the Worker and Archivist workstation. Archiving and retrieval operations require file movement so the highest bandwidth medium is preferred. Consider path mappings and thoroughly test if you need to support Windows and Mac clients as clips imported to CatDV from Windows users will require path mappings in some Worker actions. CatDV supports the use of proxies, which are not affected by archive or backup operations.

• Ensure the CatDV Worker is installed and licensed. It is recommended that the Worker folder be installed to the /Applications folder of the Mac. Configure the Worker to connect to CatDV Server where the login account has sufficient permissions to access/modify any clip in the system considered for archive or

of 26

retrieval. Please refer to CatDV Worker documentation for more information on how to install and configure the product. You may wish to place your own Worker scripts for other tasks on the same machine; this can be carried out at any time, as CatDV-StorNext Archive Additions will add archive tasks to the worker during installation.

5 Configuring StorNext A Storage Manager policy must be created to provide a folder as the target for archive operations. This policy folder will contain stub files relating to the original media. StorNext Manager is responsible for committing the selected files to long-term storage and releasing the space occupied on the SAN. CatDV-SN Archive tracks the location of media through disk to tape by updating the CatDV Server records. CatDV-SN Archive manages the retrieval process from request to restoration to the CatDV recorded path (or path equivalent) with the original file permissions. The policy folder is best located on a volume that is not normally accessed by users as this will help prevent unintended restore operations that can occur if stub files are accessed. A policy folder name that begins with a full stop ‘.’ Can also assist as the folder is not visible to standard Mac clients.

A StorNext configuration example will be shown and should act as a guide for your deployment. Whilst this section accounts for a large proportion of this manual, it is prerequisite to a functioning archive system.

5.1 SN API Password The StorNext Web Service is used for a number of media operations. This service is protected by a unique password, and saved to a file on the StorNext metadata controller. To check if the file exists, open 'Computer’ from the 'Places' menu heading. Continue opening 'Filesystem' to the path /usr/adic/ - select 'View"-'Show Hidden Files' in this directory and look for a file named .snapipassword

You may open this file in a text editor to read any password set, or type in your preference.

If the .snapipassword file does not exist on your system, the following guide will help in creating the entry.

Log on to the Red Hat Server metadata controller as root and launch ‘Text Editor’ from Applications – Accessories menu.

of 26

Figure 2 Open Text Editor

Type in your preferred password and save the text file as .snapipassword in /usr/adic.

The password you have discovered or created is needed later by the CatDV-SN Archive, so make a note.

If your StorNext installation incorporates high availability (HA) then this modification can be made to the partner server, however CatDV-SN Archive will not automatically failover and follow controller actions, please contact support if the failover situation is prolonged.

5.2 Policy Folder It is assumed that a tape library has been configured for use with the StorNext system and tapes allocated for archive operations. Please refer to StorNext documentation for help on this topic. A StorNext policy folder is required as a target for archive operations, there may be such a folder already configured but the following may help in revealing or creating a policy.

The StorNext policy folder is prerequisite to any configuration. This is achieved in the StorNext web manger, which is normally accessible from any LAN workstation. Open a web browser and use the address HTTPS://StorNext_server_name or IP_address.

The default login credentials are

User: admin

Password: password

The following system has a tape library configured

of 26

Figure 3 Storage Destination Tape Library

A view of Reports – Media. Tapes are available

Figure 4 Media Reports

5.3 Storage Policy Select Configuration – Storage Policies (A default backup policy exists)

Figure 5 Storage Manager Policies

Select ‘New’ and enter a name of ‘catdv’. The following screen shots show

of 26

suggested settings. Please refer to StorNext documentation for a more detailed explanation of the options available.

Configuration settings entered for the catdv policy – Note 5 Minutes is the minimum allowed.

Figure 6 Storage Manager Policy General Tab

The next step is to associate a directory with the policy, select the ‘Associated Directories’ tab.

This example uses a folder named .archive_catdv, created in volume ‘store3’, which must be a StorNext managed volume.

The path used here is /stornext/store3/.archive_catdv (Note the preceding ‘.’ to the folder name) This renders the folder invisible in the Finder of Macintosh clients. This is helpful in preventing inadvertent user actions on a system sensitive area.

of 26

The policy is applied to the folder you specify here. If the folder has been created on the file system then select and ‘Add’, otherwise use the ‘Create Directory’ option and ‘Add’.

Figure 7 Storage Manager Policy Associated Directories Tab

Make a note of the policy folder path as it will be required during the configuration of CatDV-SN Archive.

This completes the StorNext preparation, now is the time to choose the CatDV Worker node to handle CatDV-SN Archive requests.

6 Worker Preparation The Worker may exist on the CatDV Server (Macintosh) or a connected Macintosh client. It will need access to the media for relocation purposes and fiber interfaces provide the fastest performance. Install CatDV Worker following the default recommendations and apply your license. Configure the connection to CatDV Server, remember to use an account with sufficient permissions to read/write all clips that will

of 26

be subject to archive functions.

6.1 Install Archive Additions Install CatDV-SN Worker Additions, administrator rights will be required during installation and a user logout will occur. The installation will create a folder named SN_PROD421 at the root of the system disk containing programs, configuration and log files for the archive solution.

6.2 Create a Worker Folder Share Archivists have a command named ‘StorNext Status’ in CatDV Pro Tools menu that will display historic throughput of archive related events. The Worker hosts the application and data that must be accessible from the Archivists workstations. The following guide assumes OSX workstation is in use; please refer to OSX Server documentation for file sharing if your operating system differs.

Create a new standard user account on the Worker named SNArchive with a password of SNArchive. This will be used to restrict access to Archivists.

A folder named /SN_PROD421/Results exists in the Worker system disk, which requires sharing over AFP - SMB is also required if any CatDV Pro clients are running Windows.

Open system preferences – Sharing and select ‘File Sharing’. Add a new share and browse to /SN_PROD421/Results

Add the user SNArchive to the Results share and give read access.

Select ‘Options’ to modify protocols.

7 CatDV StorNext Archive Additions: Server Configuration

7.1 License Files CatDV-StorNext Archive Additions requires a licence for operation. The customer receives one or more files which must be copied to /SN_PROD421/Licence folder on the Worker to activate the archiving components. The license details may then be viewed in the License tab window of the CatDV-SN Archive Settings utility.

The Licence tab displays the CatDV-SN Archive product license.

of 26

Figure 8 CatDV_SN Archive Licence Tab

Note the above displays an expiry date which is ignored as the license Status is ‘Full’. A Status of ‘Demo’ will be displayed for time limited licenses.

7.2 Settings Locate and launch /SN_PROD421/bin/Settings on the CatDV Worker. This utility is used to configure the product to suit your environment and view the installed product licence. The following screen shots are taken from a working deployment, with descriptions of the required information and use.

of 26

Figure 9 CatDV_SN Archive Settings Tab

StorNext Web Service url

E.g. http://StorNext_controller_ip_address:81/axis2/services/stornext/

Substitute your active StorNext controller IP address for StorNext_controller_ip_address

Web Service pwd

Enter the characters you discovered or created in /usr/adic/.snapipassword

Policy Folder

Enter the policy folder path you created in the form:

/SAN_volume_name/policy folder name e.g. /store3/.catdv_archive

CatDV Executable

Enter the path to your CatDV Worker folder in use followed by /catdv

e.g. /Applications/CatDV Worker 5.0.3/catdv

Async WS Delay

This should remain at the default of 600

SSH IP Address

of 26

Enter your active StorNext controller IP address

SSH User ID

root

SSH Password

Enter the root account password

Days to keep logs

The archive solution produces several log files, grouped on a daily basis. Items older than the value you enter here will be discarded.

Worker Location

Enter the path to the CatDV Worker in use.

Any changes made in the Settings window are not committed until Save is selected under the File menu.

7.3 Deploying Worker Actions and Testing Communication The tab labelled ‘Pre Flight’ is used to verify communication based on your settings and to deploy the required Worker Actions for archive functions. Note the Pre Flight is run against a ‘Saved’ configuration in the ‘Settings’ tab so do not make a change to Settings without saving or Pre Flight may not give the results expected.

Figure 10 CatDV_SN Archive Pre Flight Tab

Web Service and SSH communication to StorNext is attempted when ‘Run Pre Flight’ is selected. If success is not reported, please revise your entries on the Settings tab and save prior to re-launching Pre Flight. The Worker Actions are queried for the existence of CatDV-SN Archive functions and inserted if not found.

This utility may be run at anytime to verify program integrity or apply changes, please remember to ‘Save’ any changes made in ‘Settings’ before running a ‘Pre Flight’.

of 26

Launch CatDV Worker and verify that Worker actions named SN_Archive and SN_Retrieve exist. You may elect to add more Worker tasks but take care not to accidently modify SN_Archive and SN_Retrieve when viewing. Please refer to appendix A for details of the Worker scripts, which may be useful for future comparison.

CatDV stores media path information as metadata and is used throughout the archive and retrieve processes. If your workflow requires the support of Windows media paths, CatDV-SN Archive supports the path equivalence model used by CatDV. This is best achieved on a CatDV Pro client to establish the required path mappings by testing prior to inserting into the SN_Archive action.

Note that SN_Retrieve does not require any path mapping entries.

7.4 Additional Configuration Settings When configuration changes are ‘Saved’ with the Settings utility, they are committed to a file named stornext.cfg which resides in /SN_PROD421/config folder. The Setting GUI does not expose all of the entries that exist in stornext.cfg

When volumes are created in StorNext, the user may elect to not use the offered default mount point which starts with /stornext and a manual change to stornext.cfg will be necessary if this is the case in your deployment. TextEdit may be used on the Mac Worker to change line 13 (/stornext) to your specific mount point and save the file. Please do not modify any other settings or remove blank lines as errors will result in program flow.

7.5 CatDV-‐StorNext Archive Additions Folder Structure The installation of CatDV-SN Worker Additions created a folder named SN_PROD421 at the root of the Worker system disk. The folder structure and contents should not be re-located, as archive operation will be affected.

/SN_PROD421/bin and Lib sub directory

This folder contains executable routines to support archive functions. The ‘Settings’ application is placed here, please create an alias if you wish to launch the utility from any other location.

/SN_PROD421/config

A file named stornext.cfg exists which is updated by changes made in the Settings utility. Do not modify this file directly.

/SN_PROD421/licence

You will normally receive a product licence as a zip attachment to an email. Extract the contents of the zip file to this folder in order to activate CatDV-SN Archive.

of 26

/SN_PROD421/Report

This folder contains an application named results.jar and a data file named results.xml. This folder should be shared to archivists for access to the StorNext Status utility. Please refer to Create a Worker Folder Share for details.

/SN_PROD421/logs

Log files are produced relating to phases of operation in archive functions, the files are closed at midnight and a new log is started. Some log entries contain debugging information but many report actions attempted with success or failure.

The log file naming has the following format where log_type is a description of the activity.

Log_typeyear-month-day.log

archiveyear-month-day.log – example archive2012-6-18.log

This log reports activity of the Worker detection process for archive requests.

snretrieveyear-month-day.log – example snretrieve2012-6-15.log

This log reports activity of the Worker detection process for restore requests.

checkarchiveyear-month-day.log – example checkarchive2012-6-16.log

Background process monitor for archive.

checkretrieveyear-month-day.log example checkretrieve2012-6-11.log

Background process monitor for retrieve.

execyear-month-day.log – example exec2012-6-12.log

This log reports activity of the post restore process.

of 26

8 Install the CatDV Client Additions An installation of CatDV-StorNext Archive Additions is required on the CatDV Pro workstations that require archive functions. Installers are supplied for Mac and Windows platforms that will add archive tools to an existing CatDV Pro application.

Launch CatDV Pro after installing CatDV-SN Archive Additions (aka Archivist Additions) and the Tools menu will contain extra selections similar to the following:

Please refer to CatDV-StorNext User Guide for details on use.

of 26

9 CatDV StorNext Archive Additions Worker Actions This section describes the worker actions deployed as part of the Archive Additions

9.1 CatDV-‐StorNext Archive Two actions are automatically added during the Pre Flight process and details of settings are in the following screen shots. Pre Flight only checks for the existence of the scripts, not their accuracy, so the following may be useful in case of accidental modification. Only areas which require selection or input are shown unless otherwise stated.

The worker actions are described below:

General

Figure 11 SN_Archive General Tab

Server Query

Figure 12 SN_Archive Server Query Tab

Note: Text entered is case sensitive unless the 'Ignore case' check box is ticked

of 26

Parameters

Figure 13 SN_Archive Parameters Tab

Note if Windows paths are to be supported, enter mappings into ‘Path Mappings’

Conversions

Figure 14 SN_Archive Conversions Tab

The text for ‘Execute command 1:’ is as follows and is case sensitive –

java -jar /SN_PROD421/SNmovefile.jar $b $I

of 26

Post Processing

Figure 15 SN_Archive Post-Processing Tab

Note: The ‘Publish to server’ must be set as 'Don't save changes'

Select ‘OK’ to save the action and create the second named SN_Retrieve

9.2 CatDV-‐StorNext Retrieve The worker action is described below

General

Figure 16 SN_Retrieve General Tab

of 26

Server Query

Figure 17 SN_Retrieve Server Query Tab

Note: Text entered is case sensitive unless the 'Ignore case' check box is ticked

Parameters

Figure 18 SN_Retrieve Parameters Tab

Note: The ‘Check interval’ may be lowered for testing purposes

Conversions

Figure 19 SN_Retrieve Conversions Tab

The text for ‘Execute command 1:’ is as follows and is case sensitive –

java -jar /SN_PROD421/SNretrieve.jar $b $l ${MS}

of 26

Post-Processing

Figure 20 SN_Retrieve Post-Processing Tab

Note: The ‘Publish to server’ must be set as 'Don't save changes'

of 26

10 Troubleshooting

All services must be functional to process archive functions. CatDV Server must accept the archive/retrieve requests so server login is required. The Worker Actions must be running, if metadata fields do not appear when a requested clip record is refreshed then check the Worker status. The Pre-Flight utility may be used to provide some guidance on communication or configuration problems.

There is little room for error in the CatDV Archive Settings entered and committed during the Pre Flight process. If Pre Flight does not report errors, it is worthwhile submitting an archive of test assets to the intended archive via the StorNext Digital Archive User Interface. Whilst this action bypasses a lot of automation, it will convince you that StorNext has access to the media and submissions to your archive are successful. Check StorNext for any problems with this request.

Remember to ‘Save’ any modifications you make in Settings before running ‘Pre Flight’.

When the CatDV Worker triggers a backup or archive action, the file paths are determined by path mapping associated with the event. Each entry relates to the path to be used by the Worker and entered into Archive Status.

/Applications/CatDVArchive/Logs/Service.log recycles at 5MB and a new one is generated. Logs are maintained up to a count of twenty.

The installation of CatDV Archivist Additions places additional files into a general Java extensions folder of the CatDV Pro client. This should allow for future upgrades to CatDV.

As the archive function includes a final deletion of the online file, the Worker must have the ability to delete items on the storage location indicated by CatDV media path (or equivalent path if in Windows format). If the Worker has the volume mounted as read only then the file deletion will not be successful. This can occur in some SAN storage environments where read/write access is only given to a single user at any one time.

CatDV Pro on the Windows platform may exhibit a file lock by merely having the clip selected. This can be avoided by modifying CatDV Preferences to ‘Avoid pre-loading movie’ as shown.

of 26

CatDV StorNext Archive Additions does not prevent subsequent backup/archive requests, if so desired. The validity checks will report if an operation cannot be completed due to the current archive status of the clips involved. E.g. it is not possible to restore clips, which have not yet been archived, or attempting to archive a clip that is currently offline.

If media is imported to CatDV via a Macintosh but archived by a Windows client, (or vice versa) a special case exists where this condition is discovered during media checks:

The archivist may Submit the request as normal providing path equivalence mappings are valid in Worker tasks (‘Move To Archive’ and ‘Copy To Archive’).

‘CatDV StorNext Status’ requires the archivist to enter the Worker IP address and credentials at the initial launch. Subsequent use of the tool relies upon a locally saved settings file in ~/Library/Application Support/CatDVArchive (Mac) or ~\appdata\local\CatDVArchive (PC). This file should be deleted if the target Worker or its IP address ever has to change.

catdv stornext archive additions: installation and ... · • the catdv stornext archive additions...

Documents