solaris - giovanni mantengoli cluster file system 3... · command failures ... list of cfs agents...

VERITAS Cluster File System™ 3.4

Installation and Configuration Guide

Solaris

August 200130-000383-011

Disclaimer

The information contained in this publication is subject to change without notice.VERITAS Software Corporation makes no warranty of any kind with regard to thismanual, including, but not limited to, the implied warranties of merchantability andfitness for a particular purpose. VERITAS Software Corporation shall not be liable forerrors contained herein or for incidental or consequential damages in connection with thefurnishing, performance, or use of this manual.

Copyright

Copyright © 2001 VERITAS Software Corporation. All Rights Reserved. VERITAS,VERITAS SOFTWARE, the VERITAS logo, Business Without Interruption, VERITAS TheData Availability Company, and VERITAS File System are trademarks or registeredtrademarks of VERITAS Software Corporation in the U.S. and/or other countries. Otherproduct names mentioned herein may be trademarks or registered trademarks of theirrespective companies.

Printed in the USA, August 2001.

VERITAS Software Corporation350 Ellis St.Mountain View, CA 94043Phone 650–527–8000Fax 650–527–2901http://www.veritas.com

http://www.veritas.com

Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Document Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

Getting Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Preinstallation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xii

Obtaining a License Key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xii

Chapter 1. Installing VERITAS Cluster File Systems . . . . . . . . . . . . . . . . . . . . . . . . . .1

Hardware Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Software Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Cluster Server Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Cluster File System and Volume Manager Packages . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Other Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Installation and Configuration Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Setting PATH and MANPATH Environment Variables . . . . . . . . . . . . . . . . . . . . . . . 4

Hardware Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Shared Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Fibre Channel Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Cluster Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Installing VCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Installing License Keys for VCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Preparation for Running the installvcs Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Using the VCS Interactive Installation Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Verifying VCS Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

iii

/etc/llthosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

/etc/llttab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

/etc/gabtab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

VCS Configuration File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Verifying LLT, GAB, and Cluster Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Verifying LLT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Verifying GAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Verifying Cluster Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Potential Problems Installing VCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Incorrect Permissions for Root on Remote System . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Inaccessible System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

VCS Files From a Previous Installation Exist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

VCS, LLT, or GAB Process Running . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Temp Directory Not Accessible on Local System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Invalid Cluster Name Causes Stale Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Installing CFS Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

VERITAS Patches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Installing a CFS License Key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

The VCS Web Console, Java Console, and VMSA GUIs . . . . . . . . . . . . . . . . . . . . . . . . . 27

Accessing the VCS Web Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Installing the VCS Java Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Installing the VCS Java Console on Solaris . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Installing the Java Console on a Windows Workstation . . . . . . . . . . . . . . . . . . . 28

Starting VMSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Installing the VMSA GUI on a Windows Workstation . . . . . . . . . . . . . . . . . . . . 29

Configuring and Initializing VxVM Cluster Functionality . . . . . . . . . . . . . . . . . . . . . . . 29

Configuring VxVM Using the vxinstall Command . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Installing CVM and CFS Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Running the CVM Agent Configuration Script . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Running the CFS Agent Configuration Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

iv Cluster File System Installation and Configuration Guide

Creating a Shared Disk Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Creating a Shared Volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Creating a Cluster File System and Mounting It in Shared Mode . . . . . . . . . . . . . . . . . 36

Adding a Node to an Existing Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Configuring VCS on the New Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Installing the CFS Packages on the New Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Configuring CFS and CVM Agents on the New Node . . . . . . . . . . . . . . . . . . . . . . . 40

Potential CFS Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Command Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Mounting and Unmounting File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Chapter 2. Deinstalling or Upgrading VERITAS Cluster File Systems . . . . . . . . . . .43

Deinstalling the CFS Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Deinstalling VCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Deinstalling VCS Using the uninstallvcs Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Deinstalling VCS Using pkgrm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

CFS Online Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Appendix A. VERITAS Cluster File System Agents . . . . . . . . . . . . . . . . . . . . . . . . . .53

List of CFS Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

VCS Cluster Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Service Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Modifying the Agents and Their Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Resources and Service Groups for File System Cluster Functionality . . . . . . . . . . . . . . 55

Resource and Service Group Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Dependency Ordering for CVM Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Activating CVM in the VCS Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

The vxcvmconfig Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Activating CFS in the VCS Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Contents v

The vxcfsconfig Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

The vxcfsunconfig Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Guidelines for Creating Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Creating a Service Group for a Disk Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Creating a Resource Instance for a Disk Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Creating a Resource Instance for a Shared Volume . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Creating a Resource Instance for a Cluster File System Mount . . . . . . . . . . . . . . . . 59

Example main.cf File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Example CVMTypes.cf File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Example CFSTypes.cf File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

CFSMount Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Type Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Sample Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

CFSfsckd Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Type Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67


CVMCluster Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Type Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68


CVMDiskGroup Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Type Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70


CVMMultiNIC Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Type Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72


CVMVolume Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Type Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73


Appendix B. Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

vi Cluster File System Installation and Configuration Guide

Fibre Channel Switch Example Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Cluster Platform Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79

Contents vii

viii Cluster File System Installation and Configuration Guide

Preface

IntroductionThis guide describes how to install and upgrade the components of the VERITASCluster File System to configure up to a 16-node cluster file system. The Cluster FileSystem is the clustering functionality of the VERITAS File System™ (VxFS™).

Topics covered in this preface includes:

◆ Document Organization

◆ Getting Help

◆ Conventions

◆ Preinstallation Instructions

◆ Obtaining a License Key

ix

Document Organization

Document Organization◆ Chapter 1, “Installing VERITAS Cluster File Systems,” provides an overview of the

hardware and software required for creating a cluster file system environment, anddescribes how to install and configure the software components.

◆ Chapter 2, “Deinstalling or Upgrading VERITAS Cluster File Systems.” providesinstructions on how to deinstall and upgrade CFS.

◆ Appendix A, “VERITAS Cluster File System Agents,” describes the operation of theagents required by the VERITAS Cluster Server™ to implement cluster file system andVERITAS Volume Manager™ cluster resources.

◆ Appendix B, “Hardware Configuration,” provides some example configurationsetups.

Getting HelpFor assistance with any of the VERITAS products, contact VERITAS Technical Support:

◆ U.S. and Canadian Customers: 1-800-342-0652

◆ International: +1-650-527-8555

◆ Email: [email protected]

For license information:

◆ Phone: 1-650-527-4265


◆ Fax: 1-650-527-8428

For software updates:


For additional information about VERITAS and VERITAS products, visit the website at:


For software updates and additional technical support information, such as TechNotes,product alerts, and hardware compatibility lists, visit the VERITAS Technical Supportwebsite at:

http://support.veritas.com

x Cluster File System Installation and Configuration Guide

http://support.veritas.com


Conventions

Conventions

Typeface Usage Examples

monospace Computer output, files,directories, software elementssuch as command options,function names, and parameters

Read tunables from the/etc/vx/tunefstab file.

See the ls(1) manual page for moreinformation.

monospace(bold)

User input # mount -F vxfs /h/filesys

italic New terms, book titles,emphasis, variables replacedwith a name or value

See the User’s Guide for details.

The variable ncsize determines thevalue of...

Symbol Usage Examples

% C shell prompt

$ Bourne/Korn/Bash shellprompt

# Superuser prompt (all shells)

\ Continued input on thefollowing line; you do not typethis character

# mount -F vxfs \/h/filesys

[ ] In a command synopsis, bracketsindicates an optional argument

ls [ -a ]

| In a command synopsis, avertical bar separates mutuallyexclusive arguments

mount [ suid | nosuid ]

blue text Indicates an active hypertext link In PDF and HTML files, click on linksto move to the specified location

Preface xi

Preinstallation Instructions

Preinstallation InstructionsBefore installing the Cluster File System components:

◆ Obtain a license key (see “Obtaining a License Key” on page xii for details).

◆ Ensure that the directory /opt exists and has write permissions for root.

◆ Confirm that your system has enough free disk space. The following table shows theapproximate disk space usage by directory for the VERITAS packages required tosupport cluster file system functionality. The /tmp directory is cleared after theinstallation.

Obtaining a License KeyThe VERITAS Cluster File System is a licensed product. Its constituent componentsrequire two separate license keys: one for the Veritas Cluster Server (VCS) and one for allother packages. Before you install the various components, obtain both a CFS and VCSlicense key from VERITAS. To obtain an evaluation license key, contact your salesrepresentative. If you have an evaluation license key, you must obtain a permanent licensekey when you purchase the product.

Note The vxlicense command is provided in a separate package named VRTSlic. Youmust install the VRTSlic package for the CFS licensing process to work.

VCS does not use the VRTSlic command and does not require the VRTSlicpackage. The licensevcs command and installvcs script are provided in thepkgs directory on the VERITAS product CD to install the VCS license. TheVERITAS CD must be mounted and accessible to install the VCS license.

A License Key Request Form is included when you purchase the product. If you do nothave a License Key Request Form, contact your sales representative. To obtain a licensekey, complete the License Key Request Form and send it to VERITAS by fax at the numberlisted under “Getting Help” on page x. VERITAS returns a license key by fax or email.

Directory: / /usr /opt /var /tmp

Size: 140 MB 45 MB 145 MB 15 MB 100 MB

xii Cluster File System Installation and Configuration Guide

Obtaining a License Key

Note You must obtain your license keys before installing or upgrading the Cluster FileSystem and VCS packages. Be sure to record your license keys somewhere otherthan the systems on which they are installed so that you can find them if the systembecomes inoperable or the product requires reinstallation.

You must provide your system’s host ID and machine type on the License Key RequestForm. To determine the host ID of your system, run either of the following commands:

# /usr/sbin/sysdef -h# hostid

To determine the machine type, run either of the following commands:

# /usr/sbin/prtconf | head# uname -i

During installation, you are prompted for the license key. If the software is alreadyinstalled and you are upgrading or replacing an evaluation license with a permanentlicense, follow the instructions that accompany the license key to replace the old licensewith the new one.

If you have questions concerning licensing, contact VERITAS at one of the phone numberslisted in “Getting Help” on page x.

Preface xiii

Obtaining a License Key

xiv Cluster File System Installation and Configuration Guide

Installing VERITAS Cluster File Systems
1 This chapter describes how to install and initialize up to a sixteen-node VERITAS clusterfile system. VxFS with cluster functionality (CFS) requires several VERITAS softwarepackages to configure a cluster and to provide messaging services. These packagesinclude the VERITAS File System package, the VERITAS Cluster Server (VCS) to monitorsystems and application services, VERITAS Low Latency Transport (LLT) and VERITASGroup Membership and Atomic Broadcast (GAB) for communication and messaging, andthe VERITAS Volume Manager (VxVM) to create the shared volumes necessary for clusterfile systems. Topics covered in this chapter include:
◆ Hardware Overview

◆ Software Components

◆ Hardware Setup

◆ Installing VCS

◆ Using the VCS Interactive Installation Utility

◆ Verifying VCS Installation

◆ Verifying LLT, GAB, and Cluster Operation

◆ Potential Problems Installing VCS

◆ Installing CFS Packages

◆ VERITAS Patches

◆ Installing a CFS License Key

◆ The VCS Web Console, Java Console, and VMSA GUIs

◆ Configuring and Initializing VxVM Cluster Functionality

◆ Creating a Cluster File System and Mounting It in Shared Mode

◆ Adding a Node to an Existing Cluster

◆ Potential CFS Problems

1

Hardware Overview

Hardware OverviewVxFS cluster functionality runs optimally on a Fibre Channel fabric. Fibre Channeltechnology provides the fastest, most reliable, and highest bandwidth connectivitycurrently available. By employing Fibre Channel technology, CFS can be used inconjunction with the latest VERITAS Storage Area Network (SAN) applications to provide acomplete data storage and retrieval solution. See “Hardware Configuration” on page 75for more information on hardware configuration).

The figure below shows the configuration of a cluster file system on a Fibre Channel fabricwith a disk array.

Four Node VxFS Cluster Built on Fibre Channel Fabric

Public Network

Fiber Optic Private LAN

Disk Array

Fibre Channel Switch

SCSI or Fiber Optic

PrivateNetworks

Public Network

2 Cluster File System Installation and Configuration Guide

Software Components

Software ComponentsThe software packages listed below are required for implementing cluster file systemfunctionality. They are available on the VERITAS software CD and must be installed oneach node in the cluster.

Note CFS operates only on Solaris 8 (32-bit and 64-bit) operating systems. For cluster filesystem functionality to work reliably on Solaris 8, you must have Sun patch ID110383-01, 108528-02, 108901-03, 109529-06, and 111413-02 (requires SUNWsanpackage) installed.

Cluster Server PackagesThe VERITAS Cluster Server (VCS) packages are installed by the installvcs script:

◆ VRTSvcs—VERITAS Cluster Server.

◆ VRTSllt—VERITAS Low Latency Transport software and online manual pages.

◆ VRTSgab—VERITAS Group Membership and Atomic Broadcast software and onlinemanual pages.

◆ VRTSperl—VERITAS Perl required for VERITAS Cluster Server.

◆ VRTSvcsw—VERITAS Cluster Manager Web Console.

◆ VRTSweb—VERITAS Web GUI Engine.

◆ VRTSvcsdc—VERITAS Cluster Server documentation.

Cluster File System and Volume Manager PackagesThese packages are installed by the cfsinstall script:

◆ VRTSvxfs—VERITAS File System with CFS software and online manual pages.

◆ VRTScfsdc—VERITAS File System documentation.

◆ VRTSvxvm—VERITAS Volume Manager.

◆ VRTSvmdev—VERITAS Volume Manager header and library files.

◆ VRTSvmsa—VERITAS Volume Manager Storage Administrator.

◆ VRTSvmdoc—VERITAS Volume Manager documentation.

◆ VRTSvmman—VERITAS Volume Manager online manual pages.

Chapter 1, Installing VERITAS Cluster File Systems 3

Software Components

◆ VRTSglm—VERITAS Group Lock Manager.

◆ VRTScavf—VERITAS Cluster Server Enterprise Agents for Volume Manager and FileSystem.

◆ VRTSlic—VERITAS licensing facility and online manual pages. This package mustbe installed to activate all licensable cluster functionality packages.

Other PackagesInstalled independently using the pkgadd command.

◆ VRTScscm—VERITAS Cluster Server Cluster Manager.

Installation and Configuration ScriptsThe VERITAS software CD contains scripts that make it easier to install packages,deinstall packages, and configure cluster components.

◆ installvcs—Adds the VCS-specific packages and configures nodes to function as acluster.

◆ uninstallvcs—Removes the VCS-specific packages.

◆ cfsinstall—Adds the CFS constituent packages.

◆ cfsdeinstall—Removes the CFS constituent packages.

Setting PATH and MANPATH Environment VariablesThe software and online manual pages comprising CFS are installed in several differentdirectories. To ensure that the commands and manual pages are accessible when you dothe installation, add the following to the PATH and MANPATH environment variables:

PATH=/opt/VRTSvxfs/sbin:/opt/VRTSvxfs/lib/vcs:/opt/VRTSvcs/bin:/opt/VRTSvmsa/bin:/usr/lib/fs/vxfs:/usr/lib/vxvm/bin:/usr/sbin:/etc/fs/vxfs:/etc/vx/bin

MANPATH=/opt/VRTS/man:


Hardware Setup

Hardware SetupThis section provides an overview of the hardware components required to run a clusterfile system.

Shared StorageEach node in the cluster must have a Fibre Channel I/O channel to access shared storagedevices. The shared storage can be one or more shared disks, disk array, or a FibreChannel Switch. Nodes can also have non-shared or local devices on a local I/O channel.The root file system is on a local device.

Fibre Channel SwitchThe primary component of the Fibre Channel fabric is the Fibre Channel switch. See “FibreChannel Switch Example Configuration” on page 75 for an example configuration.

Cluster PlatformsThere are several Sun Microsystems hardware platforms that can function as nodes in acluster file system cluster (see the SANPoint Foundation Suite HA Release Notes). Install theSolaris 8 operating environment on each node and install a Fibre Channel host busadapter to allow connection to the Fibre Channel switch (see “Cluster Platform Setup” onpage 76).

Note For a cluster to work correctly, all nodes must have the same time. If you are notrunning the Network Time Protocol (NTP) daemon, make sure the time on all thesystems comprising your cluster are synchronized.


Installing VCS

Installing VCSThe VCS installation utility, installvcs, is interactive. Using information you supply, itinstalls VCS on each node in the cluster, locally and remotely, and sets up the VCScommunication services, LLT (Low Latency Transport) and GAB (Group Membership andAtomic Broadcast). LLT provides kernel-to-kernel communication and monitors networkconnections. GAB manages cluster membership, monitors heartbeat communication ondisk and Ethernet, and distributes information to all nodes on the cluster. LLT and GABreplace the functions of TCP/IP for CFS private network communications. Thesecomponents of the VERITAS Cluster Server are essential for creating cluster file systems.

VCS failover capabilities, which enable you to monitor system and application servicesand to restart services on a different system when hardware or software fails, are notrequired for cluster file systems. For information on the full functionality of VCS, refer tothe online documents installed by cfsinstall under the /opt/VRTSvcsdc directory.

The installvcs utility does not install the VCS graphical user interface. See “The VCSWeb Console, Java Console, and VMSA GUIs” on page 27 for information about installingthe VRTScscm package.

Installing License Keys for VCSVCS is a licensed product. Each system requires a license key to run VCS. Theinstallvcs utility prompts you for a license key for each system. If you obtain a sitelicense, the same license can be installed on all systems in the cluster. See “Obtaining aLicense Key” on page xii.

For licensing when you install VCS using the pkgadd command, the licensevcscommand is provided in the pkgs directory on the VERITAS product CD. The VERITASCD must be accessible to install a license for VCS.


Installing VCS

Preparation for Running the installvcs UtilityAll systems in the cluster must be connected to the public network and each system musthave a unique host name by which it can be addressed on the public network. The localsystem from which you do the installation does not need to be a part of the cluster.

When you install VCS, you must provide the following:

◆ Dedicated private network hardware (see “Hardware Overview” on page 2).

◆ A name for the cluster; the name must begin with an alphabetic character (a-z, A-Z).

◆ A unique ID number from 0-255 for the cluster. Within the public subnet containingthe cluster, each cluster must have a unique ID. A new cluster using a duplicatecluster ID can cause existing clusters to fail.

◆ The host names of the systems in the cluster.

◆ The device names of the network interface cards (NICs) used for the private networksamong the systems.

Before you run the installation utility:

◆ Log in as superuser.

Make sure you have remote root (rsh) access to each node in the cluster from thesystem where installation is run. Create a /.rhosts file on each target node and adda line to the file specifying the full domain name of the source system:

star33.domainname.com root

◆ Make sure any packages or programs from previous installations of VCS arecompletely removed from each system in the cluster (see “Deinstalling the CFSPackages” on page 43).


Using the VCS Interactive Installation Utility

Using the VCS Interactive Installation UtilityThe utility installs VCS over the network. You must have permissions to remotely log in asroot on each of the systems on which you are installing VCS. The following procedureshows how to install a two-node cluster, but a cluster can contain up to sixteen nodes.

1. Log in as root on a system connected by the network to the systems where VCS is tobe installed. The system from which VCS is installed does not need to be part of thecluster.

2. Insert the VERITAS software CD. If Solaris volume management software is runningon your system, when you insert the CD it is automatically mounted as/cdrom/CD_name.

If Solaris volume management software is not available to mount the CDautomatically, you must mount it manually. After inserting the CD, enter:

# mount -F hsfs –o ro /dev/dsk/c0t6d0s2 /cdrom/CD_name

where c0t6d0s2 is the default address for the CD-ROM drive.

3. Change to the /cdrom directory:

# cd /cdrom/CD_name/pkgs

4. Start the VCS installation script:

# ./installvcs

5. Enter information at the prompts, making sure the cluster name begins with a letter ofthe alphabet (a-z, A-Z) and that the cluster ID is a unique number from 0-255:

Please enter the unique Cluster Name : VCSCluster2Please enter the unique Cluster ID : 2Enter the systems on which you want to install. (system names

seperated by spaces) : star33 star34



6. The install process analyzes each system before proceeding.

Analyzing the system for install.star33 ..... OKSunOS star33 5.8 Generic_105181-20 sun4u sparc SUNW,Ultra-2star34 ..... OKSunOS star33 5.8 Generic_105181-20 sun4u sparc SUNW,Ultra-2...

If the analysis reveals problems, such as insufficient disk space or the existence ofpreviously installed VCS files, the installation stops. A message indicates the problem(see“Potential Problems Installing VCS” on page 22).

7. The install process asks you to enter the license key for the local system. Enter alicense key as in the following example:

Enter the license key for star33 : 9999-9999-9999-9999-9999-999

The install process prompts you to enter license keys for each system in the clusterunless the license key you enter is a site license valid for all systems. After enteringthe license keys, the utility checks each system and applies the keys.

Checking for VCS requirements on system for install.star33 .. OK.star34 .. OK.

Note If you are reinstalling on systems in a cluster for which you have already providedlicense keys, installvcs discovers the keys and prompts you to confirm whetherto use the discovered keys or enter new ones.

Valid VCS key located on star33.Do you want to enter a new key? nUsing the license key located on star33.

Valid VCS key located on star34...

If the discovered key is for a site license, it is applied to all systems in the cluster. Ifyou want to enter a new key, answer y and enter the key at the prompt.

Note If you do not have a valid VCS license key, see “Obtaining a License Key” onpage xii.



8. The process now discovers information about the NICs (network interface cards) forsetting up the VCS private network. You will see messages similar to the following:

Discovering NICs for setting up private network links on star33.This may take some time.

Following is the list of discovered NICs:No. NIC Device1. hme:02. qfe:03. qfe:14. qfe:25. qfe:36. Other

a. Select the device files that correspond to the private network links you haveinstalled. (Note: Do not select the NIC used for the public network):

From the list above, please enter the number (the numberappearing in the No. column) of the NIC for the FirstPRIVATE network link: 2

From the list above, please enter the number (the numberappearing in the No. column) of the NIC for the SecondPRIVATE network link: 3

b. If you must enter a NIC device other than those discovered, enter the number thatcorresponds to “Other.” In this example, you enter 6.

Then, at the prompt, enter the complete path name of the device.

9. Answer the question about the remaining systems:

Do you have the same network cards set up on all systems (Y/N)?

If you answer no (N), for each of the cluster systems, one-by-one, the process promptsyou to select the network cards from a list of NICs discovered on that system, just asin step 8.

If you answer yes (Y), the install process summarizes the information you haveentered.



10. Examine the summary of information describing the cluster setup:

Summary information for VCS installation:----------------------------------------------------Cluster Name : VCSCluster2Cluster ID : 2

Private Network Setup on Systems :star33 :

Link1->qfe:0, Link2->qfe:1star34 :

Link1->qfe:0, Link2->qfe:1

Do you want to Continue with the Cluster installation(Y/N)[Y] ? Y

11. The utility configures the ClusterService group, which includes resources for the VCSCluster Manager (Web Console) server, the virtual IP address for the service group,and the VCS notification component.

Configuring Cluster Service Group.

12. If you want to configure the Web Console, answer yes (Y) to the next question. If youdo not want to configure the Web Console, answer no (N) and skip to step 13:

Do you want to configure the Cluster Manager(Web Console) (Y/N)[Y] ? Y

When prompted, provide information about the cluster’s public network connectionas in the following example:

Enter the Virtual IP address for the Web Server: 10.129.96.64Enter Subnet [255.255.255.0]: 255.255.240.0Enter the NIC Device for this Virtual IP address (public network)on star33 [hme0]:Do you have the same NIC Device on all other systems (Y/N)[Y] ? :Y

If you answer N, the script prompts you to enter the NIC device for this virtual IPaddress on each system, one by one.

See “Accessing the VCS Web Console” on page 27 if you want to use the Web Consoleto monitor your cluster.



13. The utility now prompts you to choose whether you want to set up email notificationby configuring SMTP notification or an SNMP console. Refer to the VERITAS ClusterServer Installation Guide for complete information on installing this funtionality.

14. The process installs the VCS packages on each system. This can take a few minutes foreach system:

Installing on star33.

Copying VRTSperl binaries.Installing VRTSperl Done.Copying VRTSllt binaries.Installing VRTSllt Done.Copying VRTSgab binaries.Installing VRTSgab Done.Copying VRTSvcs binaries.Installing VRTSvcs Done.Copying VRTSweb binaries.Installing VRTSweb Done.Copying VRTSvcsw binaries.Installing VRTSvcsw Done.Copying VRTSvcsdc binaries.Installing VRTSvcsdc Done.

Installing on star34.

Copying VRTSperl binaries.Installing VRTSperl Done.Copying VRTSllt binaries.Installing VRTSllt Done.Copying VRTSgab binaries.Installing VRTSgab Done.Copying VRTSvcs binaries.Installing VRTSvcs Done.Copying VRTSweb binaries.Installing VRTSweb Done.Copying VRTSvcsw binaries.Installing VRTSvcsw Done.Copying VRTSvcsdc binaries.Installing VRTSvcsdc Done.

Copying Cluster configuration files... Done.Installation successful on all systems.Installation can start the Cluster components on the following system/s.star33 star34



15. You are now prompted to start the cluster components. For a new installation, answeryes (Y).

Do you want to start these Cluster components now (Y/N)[Y] ? Y

Loading GAB and LLT modules and starting VCS on star33:Starting LLT...Starting LLT done.Start GABStart VCS...Loading GAB and LLT modules and starting VCS on star34:Starting LLT...Starting LLT done.Start GABStart VCS

16. The script prints the following message before quitting:

To manage your Cluster(s) using Cluster Manager (Java Console)add the VRTScscm package.

See “Installing the VCS Java Console” on page 28 if you want to install the ClusterManager.

The Cluster could be accessed using the URL :http://10.129.96.64:8181/vcs


Verifying VCS Installation

Verifying VCS InstallationAfter successful installation, you can inspect the contents of the configuration files thatwere installed and modified during the process. The information in the files correspondswith the configuration information you supplied.

/etc/llthosts

The file llthosts is a database, containing one entry per node, that links the LLT systemID (in the first column) with the LLT host name. This file is identical on each system in thecluster.

For example, the file /etc/llthosts contains entries that resemble:

0 star331 star34

/etc/llttab

The file llttab contains information that is derived during installation and used by thelltconfig command. After installation, this file lists the network links that correspondto the specific system. For example, the file /etc/llttab contains entries similar to:

set-node star33set-cluster 2link qfe:1 /dev/qfe:1 - ether - -link qfe:2 /dev/qfe:2 - ether - -start

The first line identifies the system. The second line identifies the cluster (that is, the clusterID you entered during installation). The next two lines, beginning with the linkcommand, identify the two network cards that will be used by the LLT protocol. The lastline starts the LLT protocol with the start command.

See the llttab(4) manual page for details about how the LLT configuration may bemodified. The directives in the llttab file must follow the order described in the manualpage.



/etc/gabtab

After installation, the file /etc/gabtab contains a gabconfig command that configuresthe GAB driver for use. The file /etc/gabtab typically contains entries that resemble:

/sbin/gabconfig -c -n 2

Where the -c option configures the driver for use and -n 2 specifies automatic seedingfor a two-node cluster. See the gabconfig(1M) manual page for more information.

VCS Configuration FileThe VCS configuration file /etc/VRTSvcs/conf/config/main.cf is created duringthe installation process. A typical file after the initial installation resembles:

include "types.cf"cluster VCSCluster2 (UserNames = { admin = "cDRpdxPmHpzS." } Administrators = { admin } CounterInterval = 5system star33system star34

group ClusterService (SystemList = { star33 = 0, star34 = 1 )AutoStartList = { star33, star34 )OnlineRetryLimit =3)

IP webip (Device = hme0Address = "10.129.96.64"NetMask = "255.255.240.0")

Process VRTSweb (PathName = "/opt/VRTSvcs/bin/haweb"Arguments = "10.129.96.64 8181")

VRTSweb requires webip



// resource dependency tree//// group ClusterService// {// Process VRTSweb// {// IP webip// }// }

After installation, the main.cf file contains the minimum information that defines thecluster and its nodes. In addition, the file types.cf, which is listed in the includestatement, defines the VCS bundled agents for VCS resources. The file types.cf is alsolocated in the directory /etc/VRTSvcs/conf/config after installation.

Notice that the cluster has an attribute UserNames. The installvcs utility creates auser, “admin,” whose password is encrypted; the password is “password”.

Refer to the VERITAS Cluster Server User’s Guide chapter on “The VCS ConfigurationLanguage.” The section “Sample Configuration Files” includes more extensive examplesof a main.cf and types.cf for UNIX systems.


Verifying LLT, GAB, and Cluster Operation

Verifying LLT, GAB, and Cluster OperationBefore verifying LLT, GAB, or the cluster, log in to any system in the cluster as root.

Verifying LLTUse the lltstat command to verify that links are active for LLT. This command returnsinformation about the links for LLT for the system on which it is typed. Refer to thelltstat(1M) manual page for more information. In the following example, lltstat -nis typed on each system in the cluster.

System 1

# lltstat -n

Output resembles:

LLT node information: NodeState Links*0 OPEN 2 1 OPEN 2

sys1#

System 2

# lltstat -n

Output resembles:

LLT node information: NodeState Links 0 OPEN 2*1 OPEN 2

sys2#

Note that each system has two links and that each system is in the OPEN state. Anasterisk (*) denotes the system on which the command is typed.



If the output of lltstat -n does not show all of the systems in the cluster or two linksfor each system, use the verbose option of lltstat. For example, typelltstat -nvv | more on a system to view additional information about LLT. In thefollowing example, lltstat -nvv | more is typed on a system in the cluster.

System 1

# lltstat -nvv | more

Output resembles:

Node State Link Status Address0 CONNWAIT

qfe0 DOWNqfe1 DOWN

*1 OPENqfe0 UP 08:00:20:93:0E:34qfe1 UP 08:00:20:93:0E:34

2 OPENqfe0 UP 08:00:20:8F:D1:F2qfe1 UP 08:00:20:8F:D1:F2

3 CONNWAITqfe0 DOWNqfe1 DOWN

Check that each system is OPEN, each link is UP, and each address is correct. If the outputof lltstat indicates otherwise, LLT is not operating. In the example shown, LLT isconfigured incorrectly. To obtain information about the ports open for LLT, typelltstat -p on any system. In the following example, lltstat -p is typed on onesystem in the cluster.

System 1

# lltstat -p

Output resembles:

LLT port information:Port Usage Cookie0 gab 0x0

opens: 0 1 3 4 5 6 7 8 9 10 11 12 13...connects: 0 1

sys1#

Note that two systems (0 and 1) are connected.



Verifying GABTo verify that GAB is operating, type the following command on each system:

# /sbin/gabconfig -a

If GAB is operating, the following GAB port membership information is returned:

GAB Port Memberships===================================Port a gen a36e0003 membership 01Port h gen fd570002 membership 01

Port a indicates that GAB is communicating, gen a36e0003 is a random generationnumber, and membership 01 indicates that systems 0 and 1 are connected.

Port h indicates that VCS is started, gen fd570002 is a random generation number, andmembership 01 indicates that systems 0 and 1 are both running VCS.

If GAB is not operating, no GAB port membership information is returned:

GAB Port Memberships===================================

If only one network is connected, the following GAB port membership information isreturned:

GAB Port Memberships===================================Port a gen a36e0003 membership 01Port a gen a36e0003 jeopardy ;1Port h gen fd570002 membership 01Port h gen fd570002 jeopardy ;1

For more information on GAB, refer to the VERITAS Cluster Server User’s Guide.



Verifying Cluster OperationTo verify that the cluster is operating, type the following command on one system:

# hastatus -summary

The output resembles:

-- SYSTEM STATE-- System State Frozen

A star33 RUNNING 0A star34 RUNNING 0

-- GROUP STATE-- Group System Probed AutoDisabled State

B ClusterService star33 Y N ONLINEB ClusterService star34 Y N OFFLINE

Note the system state. If the value is RUNNING, VCS is successfully installed andrunning on that node. The group state lists the ClusterService group, which is ONLINEon star33 and OFFLINE on star34. Refer to hastatus(1M) manual page. In theVERITAS Cluster Server User’s Guide, Appendix B, “Advanced Topics,” describessystem states and the transitions between them.

Also, on one of the systems, enter:

# hasys -display

The output resembles:

#System Attribute Value

star33 AgentsStopped 0

star33 ConfigBlockCount 54

star33 ConfigCheckSum 29776

star33 ConfigDiskState CURRENT

star33 ConfigFile /etc/VRTSvcs/conf/config

star33 ConfigInfoCnt 0

star33 ConfigModDate Wed Dec 31 23:00:00 1999



On each system, the output should be similar. For more information on thehasys -display command, refer to the hasys(1M) manual page. Also refer to thechapter in the VERITAS Cluster Server User’s Guide, “Administering VCS From theCommand Line.”

star33 DiskHbDown

star33 Frozen 0

star33 GUIIPAddr

star33 LLTNodeId 0

star33 LinkHbDown

star33 Load 0

star33 LoadRaw runque 0 memory 0 disk 0...

star33 MajorVersion 1

star33 MinorVersion 17

star33 NodeId 0

star33 OnGrpCnt 0

star33 ShutdownTimeout 60

star33 SourceFile ./main.cf

star33 SysName star33

star33 SysState RUNNING

star33 TFrozen 0

star33 TRSE 0

star33 UpDownState Up

star33 UserInt 0

star33 UserStr 0

#System Attribute Value


Potential Problems Installing VCS

Potential Problems Installing VCSYou may encounter problems installing VCS using installvcs. Refer to the followingparagraphs for typical problems and their solutions.

Incorrect Permissions for Root on Remote SystemThe permissions are inappropriate. Make sure you have remote root access permissionon each system to which you are installing.

Analyzing the system for install.star33 ..permission deniedRSH permissions not available on: star33

Suggested solution: Add a line in the file /.rhosts giving root access to thesystem running the installation script. A plus (+) character in the first line of /.rhostswill permit any remote host to access the system.

Inaccessible SystemThe system you specified is not accessible. This could be for a several reasons, includingthe system name was entered incorrectly or the system is not available over the network.

Analyzing the system for install.star33 .The specified system does not exist or is inaccessible.

Suggested solution: Verify that you entered the system name correctly; use theping(1M) command to verify the accessibility of the host.

VCS Files From a Previous Installation ExistA VCS package from a previous installation already exists.

Analyzing the system for install.star33 ..... OKSunOS star33 5.8 Generic_105181-20 sun4u sparc SUNW,Ultra-2star34 ..... ERRORVRTSllt package already installed.

Suggested solution: Run the pkgrm command and remove the package listed in theerror message.



VCS, LLT, or GAB Process RunningIf you use the installvcs utility to install VCS on a system where VCS or the utilitiesLLT or GAB are currently running, the installation stops with an error similar to:

Analyzing the system for install.star34 ..... ERRORGAB found running

This can occur if you try to reinstall VCS without completely removing VCS or its utilitiesfrom the previous installation.

Suggested solution: Deinstall VCS completely (see “Deinstalling the CFS Packages”on page 43).

Temp Directory Not Accessible on Local SystemThe installvcs utility requires access to /tmp on the local system. If the directory doesnot exist or is inaccessible because of access permissions, installvcs stops with an errorthat resembles:

Analyzing the system for install.star33 ..... ERRORThe /tmp directory is inaccessible on the local machine.

Suggested solution: Check whether the /tmp directory exists; create /tmp if it doesnot. Make sure that /tmp has the correct access permissions.



Invalid Cluster Name Causes Stale ConfigurationIf you begin a cluster name with any other character (a number, for example) other thanan alphabetical character, a-z or A-Z, the installation of VCS will appear to succeed, but allVCS configurations will report as “stale.”

You can verify that this is the problem by typing the command from the directory/etc/VRTSvcs/conf/config:

# hacf -verify .

You receive messages that report that the cluster name is specified in error. For example:

VCS:12123: Type ’_CLUSTER’ defined more than onceVCS:12095: Aborting hacf: errors encountered

Suggested solution: Change the cluster name to a valid name. This can be done byediting the newly created VCS configuration file and renaming the cluster:

1. Log on to a machine in the cluster.

2. Edit the VCS configuration file, /etc/VRTSvcs/conf/config/main.cf (see “VCSConfiguration File” on page 15). You can use vi or another text editor. Look for theline that contains the keyword “cluster” followed by the invalid name; correct thecluster name.

3. Save and close the file.

4. Enter the command:

# /sbin/hastart

VCS will restart on the system in a RUNNING state (see “Verifying Cluster Operation”on page 20 for more information).

5. Log on to another system in the cluster and perform step 4. VCS starts up, using therevised configuration file. Repeat this step on each of the remaining systems in thecluster.


Installing CFS Packages

Installing CFS PackagesAfter you install VCS and verify that it is functioning, you must install additionalVERITAS software on the nodes comprising the cluster. The cfsinstall script adds theadditional required VERITAS software packages to all the nodes in the cluster for you. Ifthere is an older version of VxFS on your cluster systems, remove the package beforerunning the cfsinstall script (see “Deinstalling the CFS Packages” on page 43). Do thefollowing procedure from any node in the cluster:

1. Log in as superuser.

2. Place the VERITAS software CD into a CD-ROM drive and move to the /cdromdirectory:


3. From the pkgs directory, run the installation script:

# ./cfsinstall

The script is interactive and requires that you respond to inquiries. The output liststhe packages and VERITAS patches being installed on each node. You can pressCONTROL-C at any time to stop the installation.

4. To check that the packages were installed, on each node enter:

# pkginfo -x | grep VRTSVRTScfsdc VERITAS Cluster File System DocumentationVRTSgab VERITAS Group Membership and Atomic BroadcastVRTSglm VERITAS Group Lock ManagerVRTSlic VERITAS Licensing UtilitiesVRTSllt VERITAS Low Latency TransportVRTSperl VERITAS Perl for VRTSvcsVRTScavf VERITAS Cluster Server Enterprise Agent for

Volume Manager and File SystemVRTSvcs VERITAS Cluster ServerVRTSvcsdc VERITAS Cluster Server DocumentationVRTSvcsw VERITAS Cluster Manager (Web Console)VRTSvmdev VERITAS Volume Manager, Header and Library FilesVRTSvmdoc VERITAS Volume Manager (user documentation)VRTSvmman VERITAS Volume Manager, Manual PagesVRTSvmsa VERITAS Volume Manager Storage AdministratorVRTSvxfs VERITAS File SystemVRTSvxvm VERITAS Volume Manager, BinariesVRTSweb VERITAS Web GUI Engine

Note that VRTSvcsdc, VRTSgab, VRTSllt, VRTSperl, VRTSweb, VRTSvcsw, andVRTSvcs were installed previously by the installvcs script.


VERITAS Patches

VERITAS PatchesThere are VERITAS software patches required for the CFS component packages. Thesepatches are installed automatically by the cfsinstall script and deinstalled by thecfsdeinstall script. The patches are in the patches directory on the VERITASsoftware CD. You can view the README files in this directory for descriptions of theincidents fixed by the patches. To view the README files:

1. Place the VERITAS software CD into a CD-ROM drive and move to the /patchesdirectory:

# cd /cdrom/CD_name/patches/patch_number_directory

2. In the selected patch_number_directory, open the README as shown in this example:

# view README.110435-05

Installing a CFS License KeyThere is a VERITAS license key that enables VERITAS File System clustering functionality.

Note This is a different license key from the one used to license VCS. See “InstallingLicense Keys for VCS” on page 6 for more information

To install the license key for VxFS cluster functionality, on each node enter a valid key as inthe following example:

# vxlicense -cPlease enter your key: 9999 9999 9999 9999 9999 9999

After you enter a valid key, the system displays text similar to:

vrts:vxlicense: INFO: Feature name: VXCLUSTERFS [190]vrts:vxlicense: INFO: Number of licenses: 1 (non-floating)vrts:vxlicense: INFO: Expiration date: No expiration datevrts:vxlicense: INFO: Release Level: 34vrts:vxlicense: INFO: Machine Class: 258193729vrts:vxlicense: INFO: Key successfully installed in /etc/vx/elm/190.vrts:vxlicense: INFO: Key successfully installed in /etc/vx/elm/80.vrts:vxlicense: INFO: Key successfully installed in /etc/vx/elm/88.vrts:vxlicense: INFO: Key successfully installed in /etc/vx/elm/68.vrts:vxlicense: INFO: Key successfully installed in /etc/vx/elm/95.

Note If you do not have a valid license key, see “Obtaining a License Key” on page xii.


The VCS Web Console, Java Console, and VMSA GUIs

The VCS Web Console, Java Console, and VMSA GUIsThe VCS Web-based Cluster Manager (Web Console) lets you monitor the cluster fromany workstation on the public network. Supported browsers are Netscape Navigator 4.0or later, or Internet Explorer 4.0 or later.

You can also administer VCS using the VCS Java-based graphical user interface, JavaConsole. After VCS has been installed, you can install the Java Console on a UNIX systemwith X-Windows, or on a Windows NT or Windows 2000 Professional workstation.

The Volume Manager Storage Administrator (VMSA) is the graphical administrativeinterface for configuring shared storage devices used for CFS.

For information about using the VCS GUIs, see the VERITAS Cluster Server User’s Guide.For information about using VMSA, see the VERITAS Volume Manager StorageAdminstrator Administratorr’s Guide.

Accessing the VCS Web ConsoleWhen VCS starts running in the cluster and the ClusterService Group comes up, the WebConsole server starts. To access the Web Console:

1. From the browser, navigate to the Web Console by entering:

http://web_gui_IP_address:8181/vcs

For example:

http://10.129.96.64:8181/vcs

The IP address is the “Cluster virtual IP address” configured into the ClusterServiceGroup.

2. On the Login screen, enter a valid user name and password. By default, theadministrator of a new installation can log in as “admin” and use “password” as apassword. For security, change your password at your earliest convenience.

3. Click Login to enter the Cluster Summary view.


The VCS Web Console, Java Console, and VMSA GUIs

Installing the VCS Java ConsoleAfter VCS has been installed, you can install the Java Console on a UNIX system withX-Windows, or on a Windows NT or Windows 2000 Professional workstation. The UNIXsystem from which you run the Java Console can be a node in the cluster or a remoteworkstation; the latter lets you administer each system in the cluster remotely. If an olderversion of the GUI is installed on your system, remove it by following the proceduresdescribed in the VCS documentation.

Installing the VCS Java Console on Solaris

1. Move to the /cdrom directory:


2. Type the command:

# pkgadd -d . VRTScscm

Installing the Java Console on a Windows Workstation

If you are installing the VCS Java Console (Cluster Manager) on a Windows NTworkstation or a Windows 2000 Professional Workstation to administer the cluster, do thefollowing:

1. Insert the VERITAS software CD into the drive on your Windows workstation.

2. Using Windows Explorer, select the CD drive.

3. Go to \ClusterManager.

4. Double-click Setup.exe.

5. The VCS InstallShield guides you through the installation process.


Configuring and Initializing VxVM Cluster Functionality

Starting VMSAThe VMSA GUI is installed by the cfsinstall script. To start the VMSA GUI, on themaster node, enter:

# /opt/VRTSvmsa/bin/vmsa &

Installing the VMSA GUI on a Windows Workstation

If you are installing VMSA on a Windows workstation, do the following:

1. Insert the VERITAS software CD into the drive on your Windows workstation.

2. Using Windows Explorer, select the CD drive.

3. Go to E:\pkgs_win32.

4. Double-click Setup.

5. The InstallShield guides you through the installation process.

Configuring and Initializing VxVM Cluster FunctionalityThe cluster functionality (CVM) of the VERITAS Volume Manager allows multiple hoststo simultaneously access and manage a given set of disks under VxVM control. A VxVMcluster is a set of hosts sharing a set of disks; each host is a node in the cluster. The nodesare connected across a network. If one node fails, other nodes can still access the disks.The VxVM cluster feature presents the same logical view of the disk configurations(including changes) on all nodes.

The following procedure is one way of configuring VxVM. For alternative methods ofconfiguring VxVM, or to install full VxVM functionality, see the VERITAS Volume ManagerInstallation Guide. Online versions of the VxVM documentation set are installed under the/opt/VRTSvmdoc directory.



Configuring VxVM Using the vxinstall CommandThe vxinstall script places all specified disks under VxVM control. These disks areautomatically placed in the default disk group, rootdg. Use the vxinstall script toencapsulate a boot disk, or if you do not want to encapsulate any disks, use the script toinitialize at least one, non-shared disk (and leave any shared disks alone) into rootdg.Later, you can use the VERITAS Volume Manager Storage Administrator (VMSA) or theVxVM command line interface to place shared disks into other disk groups. You mustplace at least one disk under VxVM control with the vxinstall script before you canadd disks using other utilities or interfaces.

It is important for correctly configuring cluster functionality that you create your rootdgwhen you run vxinstall. Run vxinstall on each node of the cluster.

1. To start the VxVM installation procedure, enter:

# vxinstall

2. The vxinstall script examines all controllers attached to the system and lists them.It then asks whether you want to do a Quick Installation or Custom Installation. Toencapsulate some disks on a given controller and initialize others, use the CustomInstallation. To allow mirroring, be sure to encapsulate your root disk.

3. Reboot the system.

4. Check the configuration of the root disk group:

# vxprint

Installing CVM and CFS AgentsAgents are processes that bring resources online and take them offline. They also monitorresources and report any state changes to VCS. There are specific agents for cluster filesystem that you must install. See “VERITAS Cluster File System Agents” on page 53 forbackground information on CFS and CVM agents.



Running the CVM Agent Configuration Script

1. From any node in the cluster, run the script to install CVM agents. Answer thequestions as prompted:

# /usr/lib/vxvm/bin/vxcvmconfig

Menu: CVM service group1 Add the Configuration for CVM Service Group2 Remove the Configuration for CVM Service Group? Display help about menu?? Display help about the menuing systemq Exit from menus

Select an operation to perform: 1

Enter 1 to proceed with the installation. You can exit the program any time byentering q, or access the online Help menus by typing ? or ??.

2. The program displays the configuration information:

The cluster configuration information read from the VCSconfiguration file is as follows.Cluster : VCSCluster2Nodes : star33 star34

You will now be prompted to enter the information pertainingto the cluster and the individual nodes.Hit RETURN to continue.

Press RETURN to continue.

3. The program prompts you for daemon port numbers. These port numbers are used byCVM for configuration level and kernel level communication between two nodes inthe cluster:

Port number for vxconfigd: [<port_number>,q,?] (default: 4500)Port number for vxkmsgd: [<port_number>,q,?] (default: 4501)

Use the defaults unless port 4500 and 4501 are in use.

4. The program prompts you for the timeout value in number of seconds. The timeout isused by CVM during cluster reconfiguration. The appropriate value depends on thenumber of nodes in the cluster and the size of the shared disk group configuration.Use a larger value for larger configurations. In most cases, the default value of 200seconds is sufficient:

Timeout: [<timeout>,q,?] (default: 200)



5. Enter the number of private networks:

Number of private networks : [<num>,q,?] 2

6. Enter the type of private network NICs (network interface cards) used by each nodein the cluster. Specify the same interfaces as shown under the link directive in the/etc/llttab file. The order entered is the order in which the corresponding privatenetwork is selected by the CVMMultiNIC agent to failover the CVM IP address.

The program will prompt you for each node. Specify one interface per network, andin the same order for all nodes:

What is the network interface on network 0? [<nic>,q,?] qfe0What is the network interface on network 1? [<nic>,q,?] qfe1

Enter the interface in the format interface name followed by the physical unit number.For example, use qfe0, not a full device path name such as /dev/qfe:0.

7. Specify IP addresses for the network interfaces. CVM uses virtual IP addresses forcommunication and to fail over from another node if there is a fault on the activeprivate network. If you do not specify the addresses, the CVMMultiNIC agentconfigures the base IP addresses on the physical interfaces when the CVM servicegroup is later put online (the default IP address used is 169.254.x.x).

Do not enter the same address for the base and virtual IP address. Enter addressesbelonging to the same subnet in standard Internet dot notation.

Would you like to specify the IP addresses? [y,n,q,?] n

Note If the private network is shared, make sure there are no conflicting IP addresses onthe shared network.

See the online Help and the ifconfig(1M) manual page for more information.



8. The program then prints a summary of the configuration information similar to thefollowing:

------- Following is the summary of the information: ------Cluster : VCSCluster2Nodes : star33 star34vxconfigd port : 4500kmsg port : 4501Timeout : 200

-------------------------------------------------------------------------- Information for Node star33:------------

NodeID : 0 Virtual IP Addr : 169.254.65.1 Base IP Addr for qfe0 : 169.254.65.101 Base IP Addr for qfe1 : 169.254.65.102

-------------------------------------------------------------------------- Information for Node star34:------------

NodeID : 1 Virtual IP Addr : 169.254.65.2 Base IP Addr for qfe0 : 169.254.65.103 Base IP Addr for qfe1 : 169.254.65.104

-----------------------------------------------------------

The program prompts whether the summary is correct. If you enter y, the main.cffile is updated with the CVM configuration information. Before updating, the originalmain.cf is copied to /etc/VRTSvcs/conf/config/main.cf.save and a newCVMTypes.cf is created. If one already exists in /etc/VRTSvcs/conf/config, itis moved to CVMTypes.cf.date.

If the information is not correct, enter n and the program exits. Rerun vxcvmconfigwith the correct information.

9. To end the procedure, you can put the CVM service groups on the systems in thecluster online:

Do you want to online cvm service group ? [y,n] y

If you enter n, you can put the service group online later using the VCS GUI or theVCS command line interface.



Running the CFS Agent Configuration Script

1. From any node in the cluster, run the script to install the CFS agents on all nodes:

# /opt/VRTSvxfs/lib/vcs/vxcfsconfig

2. After the agents are completely installed, check to see CVM status. On each node,enter:

# /usr/lib/vxvm/bin/vxclustadm nodestatestate: cluster member

3. On each node in the cluster, determine whether the node is the master or slave andwhether it’s active:

node2# vxdctl -c modemode: enabled: cluster active - MASTER

node1# vxdctl -c modemode: enabled: cluster active - SLAVE



Creating a Shared Disk Group

1. On the master node, use the VMSA GUI to create a shared disk group and add thedisks to it. To start the VMSA GUI, enter:

# /opt/VRTSvmsa/bin/vmsa &

The VMSA application will open and you can begin creating disk groups.

2. Alternatively, you can use the following script to create a new shared disk group, forexample, “cfsdg,” and add the disks to it. Fill in the name of your shared disk group,shared_dg_name, and the list of devices and controllers, shared_device_list. Ifyou want to create more disk groups, you must run this script for each.

#!/usr/bin/sh -x#export PATH=$PATH:/usr/sbin## Name of the Shared disk group#shared_dg_name="cfsdg"## List of shared devices to be part of the shared disk group#shared_device_list="c2t0d0 c2t1d0 c2t2d0 c2t3d0 c2t4d0c1t6d0 c1t8d0 c3t4d0"

first="yes"count=0

for i in $shared_device_list; do/etc/vx/bin/vxdisksetup $ivxdisk online $i"s2"vxdisk -f init $i"s2"count=‘expr $count + 1‘

if [ $first = "yes" ]; thenvxdg -s init $shared_dg_name $shared_dg_name$count=$i"s2"first="no"

elsevxdg -g $shared_dg_name adddisk $shared_dg_name$count=$i"s2"

fidone


Creating a Cluster File System and Mounting It in Shared Mode

Creating a Shared Volume

Note The following is a command-line example of creating a shared volume. It ispreferable to use the VMSA GUI to create CFS resources (see step 1 on page 35).

Create a shared volume on the CVM master. In this example, the volume is 20 MB in size:

# vxassist -g cfsdg make vol1 20M

Then check the configuration:

# vxprint

Creating a Cluster File System and Mounting It in SharedMode

On any cluster node, create a file system on the shared storage volume:

# mkfs -F vxfs /dev/vx/rdsk/cfsdg/vol1

If you do not specify size, the complete volume is used for the file system.

Mount the cluster file system individually from each node. The mount -o clusteroption mounts the file system in shared mode. Create a directory on each node on whichto mount the shared volume. It’s a good idea to give the mount points on each node thesame name.

# mount -F vxfs -o cluster /dev/vx/dsk/cfsdg/vol1 /mnt

You can add more options as a comma separated list after the -o cluster option. Allmount options must be identical on all nodes. See mount(1M) for the options compatiblewith -o cluster.


Adding a Node to an Existing Cluster

Adding a Node to an Existing ClusterIf you want to add a new node to a cluster, first prepare the new system hardware.Physically connect the new system to the cluster using private networks and attach to anyshared storage. Then install the required OS software (review the information under“Cluster Platforms” on page 5).

Configuring VCS on the New Node

1. Log in to the new system as superuser.



3. The installvcs script cannot install software on a new node, so you must usepkgadd to install all VCS packages (the VRTSvcsdc documentation package isoptional):

# pkgadd -d . VRTSperl VRTSvcs VRTSgab VRTSllt VRTSvcswVRTSweb VRTSvcsdc

4. Before installing VCS on the new node, you must run the licensevcs utility. Theutility prompts you to provide the license key (see “Obtaining a License Key” onpage xii if you have not obtained a license key for the new system). Enter the licensecommand:

# ./licensevcs

You receive a message resembling the following example:

Cannot find a VCS Cluster on this systemEnter the systems on which you want to license for VCS.(system names separated by spaces) :star35

Enter the name of the system.



5. Enter the license key as shown in the following example:

Enter the license key for star35: 9999-9999-9999-9999-9999-999

6. Create the /etc/llthosts file on the new node and modify /etc/llthosts onthe other nodes to add another node name (see “/etc/llthosts” on page 14). Followingthe previous convention in this guide, add the name star35:

0 star331 star342 star35

The /etc/llthosts files on all the nodes must be the same.

7. Create the /etc/llttab file as it looks on another node in the cluster (see“/etc/llttab” on page 14). Change the set-node line to the name of the new node:

set-node star35set-cluster 2link link1 /dev/qfe:0 - ether - -link link2 /dev/qfe:1 - ether - -start

8. Run the LLT configuration command:

# /sbin/lltconfig -c

9. Create the /etc/gabtab file as it is on another node in the cluster (see “/etc/gabtab”on page 15):

# /sbin/gabconfig -c -n 3

Note the change from n2 to n3. Update the /etc/gabtab file on the other nodesalso. There is no need to reboot the other nodes.



10. Run the GAB configuration command (see “Verifying GAB” on page 19). Make sureport A membership includes the new node.

# /sbin/gabconfig -aGAB Port Memberships===================================Port a gen a36e0003 membership 012

11. From the new node (star35), start the VCS server:

# hastart

12. On each node, run the GAB configuration command. Make sure port A and Hmemberships includes the new node.

# /sbin/gabconfig -aGAB Port Memberships===================================Port a gen a36e0003 membership 012Port h gen fd570002 membership 012

Installing the CFS Packages on the New Node

1. Follow the CFS package installation procedure as described under “Installing CFSPackages” on page 25. When prompted, install the packages only on the new node.

2. Install a license key as described under “Installing a CFS License Key” on page 26.

3. Run vxinstall to create rootdg as described under “Configuring VxVM Using thevxinstall Command” on page 30.



Configuring CFS and CVM Agents on the New NodeAfter rebooting the new system, you must configure the CFS and CVM agents on it. Usethe VCS GUI for this, or follow the command line procedure below.

1. Unmount all VxFS cluster file systems from the original nodes.

2. Take the CVM service group offline:

# hagrp -offline cvm -sys star33# hagrp -offline cvm -sys star34

3. Open the VCS configuration for writing:

# haconf -makerw

4. Add the new node to the CVM system list and specify a failover priority of zero:

# hagrp -modify cvm SystemList -add star35 0

5. Add the new node to the CVM AutoStartList:

# hagrp -modify cvm AutoStartList star33 star34 star35

6. Add the virtual IP addresses (see step 8 on page 33) for the CVM link for the newnode, which in this example is node 2:

# hares -modify cvm_clus CVMNodeId -add star35 2# hares -modify cvm_clus CVMNodeAddr -add star35 169.254.65.3

The virtual and real IP addresses for the new node and the existing nodes must be onthe same subnet.

7. Configure the real IP addresses for the private link for the new node (see step 7 onpage 32):

# hares -modify cvmmnic CVMAddress -add star35 169.254.65.3# hares -modify cvmmnic CVMDevice -add qfe0 169.254.65.105 -sys star35# hares -modify cvmmnic CVMDevice -add qfe1 169.254.65.106 -sys star35

8. Write the new VCS configuration to disk:

# haconf -dump -makero

9. Put the CVM resources back online:

# hagrp -online cvm -sys star33# hagrp -online cvm -sys star34# hagrp -online cvm -sys star35


Potential CFS Problems

10. Check the system status to see whether the new node is online:

# hastatus -sum-- SYSTEM STATE-- System State FrozenA star33 RUNNING 0A star34 RUNNING 0A star35 RUNNING 0

-- GROUP STATE-- Group System Probed AutoDisabled StateB cvm star33 Y N ONLINEB cvm star34 Y N ONLINEB cvm star35 Y N ONLINEB vxfsckd_svcgrp star33 Y N ONLINEB vxfsckd_svcgrp star34 Y N ONLINEB vxfsckd_svcgrp star35 Y N ONLINE


Command Failures

❖ Manual pages not accessible with the man command. Set the MANPATH environmentvariable as listed under “Setting PATH and MANPATH Environment Variables” onpage 4.

❖ The mount, fsck, and mkfs utilities reserve a shared volume. They fail on volumesthat are in use. Be careful when accessing shared volumes with other utilities such asdd, it is possible for these commands to destroy data on the disk.

❖ Running some commands, such as fsadm -E /vol02, can generate the followingerror message:

vxfs fsadm: ERROR: not primary in a cluster file system

This means that you can run this command only on the primary, that is, the systemthat mounted this file system first.



Mounting and Unmounting File Systems

❖ If mount fails with an error message:

vxfs mount: cannot open mnttab

/etc/mnttab is missing or you do not have root privileges.

❖ If mount fails with an error message:

vxfs mount: device already mounted, ...

The device is in use by mount, mkfs or fsck on the same node. This error cannot begenerated from another node in the cluster.

❖ If this error message displays:

mount: slow

The node may be in the process of joining the cluster.

❖ If you try to mount a file system that is already mounted without –o cluster option(that is, not in shared mode) on another cluster node,

# mount -F vxfs /dev/vx/dsk/share/vol01 /vol01

the following error message displays:

vxfs mount: /dev/vx/dsk/share/vol01 is already mounted, /vol01is busy, allowable number of mount points exceeded, or clusterreservation failed for the volume

❖ If umount fails with an error message:

vxfs umount: /vol01 cannot unmount : Device busy

You must unmount the file system on all secondary systems before unmounting itfrom the primary.


Deinstalling or Upgrading VERITAS ClusterFile Systems
2
This chapter provides information on how to deinstall CFS software or upgrade yourcluster to the latest release level. Topics in this chapter include:

◆ Deinstalling the CFS Packages

◆ Deinstalling VCS

◆ CFS Online Upgrade

Deinstalling the CFS PackagesYou can remove all remaining VERITAS cluster related packages using thecfsdeinstall script that is on the VERITAS software CD. If the deinstallation script isunavailable, you can deinstall the individual packages using the pkgrm command.

Note The VxFS package cannot be removed if any VxFS file systems are mounted.Unmount all VxFS file systems before removing the package. After you remove theVxFS package, VxFS file systems are not mountable or accessible until another VxFSpackage is installed.

1. Check if any VxFS file systems or Storage Checkpoints are mounted:

# df -F vxfs

2. First unmount and remove any Storage Checkpoints:

# umount /checkpoint_name# fsckptadm remove checkpoint_name /mount_point

Then unmount any mounted file systems.

43

Deinstalling the CFS Packages

3. Remove the VxFS and cluster related packages:

# pkgrm VRTScavf VRTSvxfs VRTScfsdc VRTSvxvm VRTSvmdev \VRTSvmsa VRTSvmdoc VRTSvmman VRTSglm VRTSlic

4. If removing the VRTSvxvm package gives the following error:

Checking for system volumes:swap ... defined on volumes: swapvol./ ... defined on volume rootvol.

The root (/) file system or other file systems are encapsulated by VxVM. Continuingwith the VxVM package removal can leave the system in an unbootable state. Youmust convert the listed volumes back into regular disk partitions before theVRTSvxvm package can be removed.

First, if any volumes have more than one plex associated with them, remove theunnecessary plexes:

# vxplex -o rm dis plex_name

Use the VxVM vxunroot (see vxunroot(1M)) utility to convert any file systemsfrom VxVM volumes to regular partitions.

# /etc/vx/bin/vxunroot

This changes the volume entries in /etc/vfstab to the underlying disk partitionsfor rootvol, swapvol, usr, and var. It also modifies /etc/system and promptsfor a reboot so that the disk partitions are mounted instead of volumes for root,swap, usr, and var.

5. Alternatively, you can run the cfsdeinstall script. Place the VERITAS software CDinto a CD-ROM drive and move to the /cdrom directory:



Deinstalling the CFS Packages

6. From the cfs directory, run the deinstallation script:

# ./cfsdeinstall

The system responds with messages similar to the following:

Removing patches on star33110436-01 110444-01 110447-01 110435-02 110707-01111118-02 111122-01Removing packages on star33Removed package VRTSvmdoc from star33Removed package VRTScfsdc from star33Removed package VRTSglm from star33Removed package VRTScavf from star33Removed package VRTSvxfs from star33Removed package VRTSlic from star33Removed package VRTSvmman from star33Removed package VRTSvmsa from star33Removed package VRTSvmdev from star33Removed package VRTSvxvm from star33Deinstallation of VERITAS cluster file system packages completed.

Note The cfsdeinstall script removes only the VCS patches, not the VCS packages. Ifyou intend to remove the VCS packages, however, you must first remove thepatches. Note also that the cfsdeinstall script allows you to select the nodesfrom which to remove the patches.

Chapter 2, Deinstalling or Upgrading VERITAS Cluster File Systems 45

Deinstalling VCS

Deinstalling VCSYou can deinstall VCS cluster components using the pkgrm command, or theuninstallvcs utility available on the VERITAS software CD. Before removing VCSfrom any system in the cluster, shutdown and remove applications such as the ClusterManager or any VCS enterprise agents that depend on VCS.

Deinstalling VCS Using the uninstallvcs Utility


2. Before running the utility, enter the following command to gracefully stop VCS:

# hastop -all



4. From the pkgs directory, run the deinstallation script:

# ./uninstallvcs

5. Answer the prompt with the names of the systems on which to deinstall VCS:

Enter the systems on which you want to uninstall. (system namesseperated by spaces) : star33 star34

Analysing the system for uninstall.star33 ..... OK.star34 ..... OK.

6. The VCS packages found on the systems are listed and removed.


Deinstalling VCS

Deinstalling VCS Using pkgrmOn each node in the cluster:

1. Shut down VCS on the local system using the hastop(1M) command.

# hastop -local

2. Stop the GAB and LLT utilities.

# gabconfig -U# lltconfig -U

3. Unload the GAB and LLT kernel modules.

a. Use modinfo and grep to determine the module IDs. The modinfo commandindicates the ID of the module in the left-hand column

# modinfo | grep GAB# modinfo | grep llt

b. Unload the modules from the kernel, using modunload.

# modunload -i ID_GAB# modunload -i ID_llt

4. In the directory /etc/VRTSvcs, use pkgrm to remove the VCS packages in thefollowing order:

# pkgrm VRTSperl VRTSvcs VRTSgab VRTSllt VRTSvcswVRTSweb VRTSvcsdc

Repeat step 1 through step 4 for each system from which to deinstall VCS.


CFS Online Upgrade

CFS Online UpgradeThis section describes how to perform an online upgrade on a multi-node VERITAScluster file system. Follow this procedure if you have a previously installed cluster. See theSANPoint Foundation Suite HA 3.4 Release Notes for information on the release levels of theCFS software components that can be upgraded.

Use one of the VCS GUIs to begin removing CFS resources, or use the command-lineprocedures below.

Note CFS operates only on Solaris 8 (32-bit and 64-bit) operating systems. For cluster filesystem functionality to work reliably on Solaris 8, you must have Sun patch ID110383-01, 108528-02, 108901-03, 109529-06, and 111413-02 (requires SUNWsanpackage) installed.


2. From any node in the cluster, make the VCS configuration writable:

# haconf -makerw

3. Select another node (in this example, node1) and enter the following command tostop VCS:

# hasys -freeze -persistent -evacuate node1

This also unmounts all cluster mounted file systems in the VCS configuration andstops CVM.

4. Remove all VERITAS software packages on the node except VRTSvcs, VRTSperl,and VRTSvxvm (including optional packages for documentation and VRTScscm):

# pkgrm VRTScavf VRTSvxfs VRTScfsdc VRTSvmdev VRTSvmsa VRTSvmdocVRTSvcsdc VRTSvmman VRTSglm VRTSlic VRTSgab VRTSllt VRTScscm

5. Reboot the node.




CFS Online Upgrade

7. Add all the new packages except VRTSvcs and VRTSperl (documentation packagesand VRTScscm are optional):

# pkgadd VRTSlic VRTSvxfs VRTSvxvm VRTSvmdev VRTSvmsa VRTScavfVRTSglm VRTSvmdoc VRTSvmman VRTScfsdc VRTSvcsdc VRTSllt VRTSgabVRTSvcsw VRTSweb VRTScscm

8. Move to the patches directory:

# cd /cdrom/CD_name/patches

9. Add the required patches:

# patchadd -M . 110435-05 110436-02 111897-01

10. Reboot the node.

11. Restart VCS:

# hasys -unfreeze -persistent node1

12. Bring the CVM group online:

# hagrp -online cvm -sys node1

Perform the above procedure on every node. After all the packages are upgraded on allthe nodes, you can begin to upgrade to the new VCS package. Up to this point, the clustermaintained its monitoring and failover capability. As you replace VCS, there is a shortwindow of time when monitoring and failover capability cannot function. See theVERITAS Cluster Server Instalation Guide for detailed information on upgrading VCS.

1. The types.cf file installed with VCS 2.0 contains new type definitions. If yourcurrent configuration altered the default types.cf file originally installed on yourcluster, make a copy of it for use later:

# cp /etc/VRTSvcs/conf/config/types.cf \/etc/VRTSvcs/conf/types.save

2. The new VCS configuration file, /etc/VRTSvcs/conf/config/main.cf, is notcompatible with the previous version. Make a copy of the current main.cfconfiguration file to edit after VCS 2.0 is installed:

# cp /etc/VRTSvcs/conf/config/main.cf \/etc/VRTSvcs/conf/config/main.save


CFS Online Upgrade

3. To begin upgrading VCS, write the VCS configuration to disk:

# haconf -dump -makero

4. Remove the configuration file to allow a new configuration to be installed:

# rm /etc/VRTSvcs/conf/config/main.cf

5. Stop VCS on all nodes:

# hastop -all -force

At this point, applications may still be running, but they are not being monitored byVCS.

6. Remove VRTSvcs and VRTSperl:

# pkgrm VRTSvcs VRTSperl

7. If he VERITAS software CD is no longer in place, insert it into a CD-ROM drive andmove to the /cdrom directory:


8. Add the new VRTSvcs and VRTSperl packages:

# pkgadd VRTSvcs VRTSperl

9. Compare the saved types.cf file (created in step 1 above) to the newly installed/etc/VRTSvcs/conf/config/types.cf:

# diff -w /etc/VRTSvcs/conf/types.save \/etc/VRTSvcs/conf/config/types.cf

If the differences include any types defined in the types.save file, edit the newlyinstalled /etc/VRTSvcs/conf/config/types.cf file, adding the types used inyour previous configuration.

10. You must run the script upgradecfg.pl to removes obsolete attributes from themain.cf file. First, set the variable VCS_HOME. If you are using the Bourne Shell (shor ksh), type:

# VCS_HOME=/opt/VRTSvcs; export VCS_HOME

If you are using the C Shell (csh), type:

% setenv VCS_HOME /opt/VRTSvcs;


CFS Online Upgrade

11. The upgradecfg.pl command uses the following syntax:

./upgradecfg.pl -i file_to_convert -o new_file_name

(You can print the usage for the command by entering the –h option.)Run the upgradecfg.pl command to remove the old attributes:

# ./upgradecfg.pl -i /etc/VRTSvcs/conf/config/main.save \-o /etc/VRTSvcs/conf/config/main.cf

12. In the main.cf file, after the UserNames attribute add the line that includes thoseusers who are to function as administrators. For example:

UserNames = { admin = "cDRpdxPmHpzS.”, joe = "eorDRerxPrzPx" }Administrators = { admin }

Note You can repeat step 9, step 10, step 11, and step 12 above on each node or simplycopy main.cf and types.cf to all the other nodes.

13. Run the licensing utility on each node so that you can enter a license key (see“Obtaining a License Key” on page xii if you do not have a valid license key):

# ./licensevcs

14. Enter the license key as shown in the following example:

Enter the license key for node1: 9999-9999-9999-9999-9999-999

15. Start VCS on each node:

# hastart

At this point, the VCS monitoring and failover capability is fully functional.


CFS Online Upgrade


VERITAS Cluster File System Agents
A Agents are processes that manage predefined resource types. When an agent is started, itobtains configuration information from the VERITAS Cluster Server (VCS). It thenperiodically monitors the resources and updates VCS with the resource status. Typicallyagents:
◆ Bring resources online

◆ Take resources offline

◆ Monitor resources and report any state changes to VCS

VCS bundled agents are part of VCS and are installed when VCS is installed. The clusterfunctionality agents are add-on resources to VCS for the VERITAS File System andVERITAS Volume Manager (VxVM). Cluster functionality agents and resource types arepart of the VRTScavf package and are added when you run the cfsinstall script. Formore information on VCS agents, see the VERITAS Cluster Server Bundled Agents ReferenceGuide. PDF and PostScript versions of this guide are located under the /opt/VRTSvcsdcdirectory.

List of CFS AgentsThe CFS agents include:

◆ CFSMount Agent

◆ CFSfsckd Agent

◆ CVMCluster Agent

◆ CVMDiskGroup Agent

◆ CVMMultiNIC Agent

◆ CVMVolume Agent

53

VCS Cluster Components

VCS Cluster ComponentsResources, attributes, and service groups are components integral to cluster functionality(see the VERITAS Cluster Server User’s Guide in the /opt/VRTSvcsdc directory for moredetailed information).

ResourcesResources are hardware or software entities, such as disks, volumes, file system mountpoints, network interface cards (NICs), IP addresses, applications, and databases.Resources work together to provide a service to clients in a client/server environment.Resource types are defined in the types.cf file by a collection of attributes. The VCSconfiguration file, main.cf, contains the values for the attributes of the resources. Themain.cf file incorporates the resources listed in the types.cf by way of an includedirective.

AttributesAttributes contain data regarding the cluster, nodes, service groups, resources, resourcetypes, and agents. A specified value for a given attribute configures the resource tofunction in a specific way. By modifying the value of an attribute of a resource, youchange the way the VCS agent manages the resource. Each attribute has a definition and avalue. You define an attribute by specifying its data type and dimension. Attributes alsohave default values that are assigned when a value is not specified.

Service GroupsService groups are comprised of related resources. When a service group is broughtonline, all the resources within the group are brought online.


Modifying the Agents and Their Resources

Modifying the Agents and Their ResourcesYou can use the VCS Cluster Manager GUI, or enter VCS commands from the commandline, to dynamically modify the configuration of the resources managed by an agent. Youcan also edit the main.cf file directly, but you must reboot your system for the changesto take effect. An example main.cf file is located in the/etc/VRTSvcs/conf/sample_cvm directory.

Because of its ease of use, it is highly recommended that you use the VERITAS ClusterServer GUI to administer your cluster file system resources. See “The VCS Web Console,Java Console, and VMSA GUIs” on page 27 for instructions on how to install the GUI, andthe VERITAS Cluster Server User’s Guide for detailed information on how to perform VCSadministrative tasks.

Resources and Service Groups for File System ClusterFunctionality

Managing cluster mounts through VCS requires various resources types, resources, andservice groups. The VCS resource types for the cluster file system (CFS) functionality areimplemented by the following agents:

◆ CFSMount

◆ CFSfsckd

The VCS resource types for the VERITAS Volume Manager cluster functionality (or CVM)are implemented by the following agents:

◆ CVMCluster

◆ CVMDiskGroup

◆ CVMMultiNIC

◆ CVMVolume

These agents are described in detail in the next section.

The installation procedure for CVM (see “Activating CVM in the VCS Environment” onpage 57) creates a VCS service group, named cvm, that contains the resource instances forCVMMultiNIC and CVMCluster. These resource instances control the overall operation ofCVM. The remaining two CVM agents, CVMDiskGroup and CVMVolume, are used whencreating service groups that control individual disk groups, volumes, and cluster mounts.

Appendix A, VERITAS Cluster File System Agents 55

Resources and Service Groups for File System Cluster Functionality

The installation procedure for CFS (see “Activating CFS in the VCS Environment” onpage 58) creates a resource instance for CFSfsckd. CFSfsckd is the VCS agent that startsthe cluster file system check (fsck) daemon, vxfsckd. The vxfsckd must be runningbefore a cluster mount can succeed.

A cluster mount on a shared volume requires a VCS resource instance for the shared diskgroup, shared volume, and cluster mount. It is advisable to create a VCS service group foreach shared disk group. Within that service group, there will be multiple VCS resources:one instance of CVMDiskGroup for the disk group; multiple instances of CVMVolume,one per volume; and multiple instances of CFSMount, one per cluster-mounted filesystem.

Resource and Service Group DependenciesDependencies between resources specify the order in which the resources within a servicegroup are brought online and taken offline. The various service groups and resourcesmust be linked so that they meet the following dependency rules:

◆ CFSMount must depend on its associated CVMVolume and on CFSfsckd

◆ CVMVolume must depend on its CVMDiskGroup

◆ CVMDiskGroup must depend on the cvm service group.

Dependency Ordering for CVM ResourcesCVM resources (see “Activating CVM in the VCS Environment” on page 57) have thefollowing dependency linkages:

◆ The CVMCluster resource requires the CVMMultiNIC resource.

◆ The CVMDiskGroup resource requires the CVMCluster resource.

◆ The CVMVolume resource requires the corresponding CVMDiskGroup.

If CVMDiskGroup and CVMVolume resources are added in a group other than the CVMservice group, that service group must have a dependency on the CVM service group asdescribed above.


Activating CVM in the VCS Environment

Activating CVM in the VCS EnvironmentUse the /usr/lib/vxvm/bin/vxcvmconfig script to dynamically add CVM-specifictypes and resources (listed under “Resources and Service Groups for File System ClusterFunctionality” on page 55) to a running VCS system. VCS must be started before you canrun the vxcvmconfig script. You only need to run vxcvmconfig from one node in thecluster to configure all nodes.

The vxcvmconfig ScriptThe vxcvmconfig script is an interactive, menu driven utility used to add or removeCVM configuration in the VCS environment. The script creates a CVMTypes.cf file in thedirectory /etc/VRTSvcs/conf/config, and updates the main.cf file. When youselect the option to add the configuration for the CVM service group, vxcvmconfigprompts you for information required by the CVMMultiNIC and CVMCluster resources,and supplies defaults where appropriate. There is a help facility at every prompt. You canenter a question mark (?) at a prompt to display a context-sensitive help message.

The vxcvmconfig script reads the name of the cluster and the nodes from the VCSconfiguration file main.cf. It prompts for the required information and adds the CVMtypes and resources: CVMCluster and CVMMultiNIC, as part of the CVM service groupto the VCS configuration. It also asks whether to put the CVM service group online oneach of the cluster nodes. Based on your choice, it puts the CVM service group on allnodes in the cluster that have VCS running online.

If you do not use the vxcvmconfig script to start the CVM service group, you must dothis from the command line at a later time. See the VERITAS Cluster Server User’s Guide forinstructions on how to manage VCS service groups and resources.

Before updating the VCS configuration with the CVM service group information (seehaconf (1M) for more information), vxcvmconfig copies the original main.cf file into/etc/VRTSvcs/conf/config/main.cf.save, so you can easily recover the originalmain.cf.

Use the VCS command line interface or the VCS GUI to add the CVMDiskGroup and theCVMVolume resources.

When you select the option to remove the configuration for the CVM service group,vxcvmconfig removes the following CVM information:

◆ CVMCluster and CVMMultiNIC resources

◆ CVMDiskGroup and CVMVolume resources belonging to the CVM servicegroup (if any)

◆ CVM service group

◆ CVMMultiNIC, CVMCluster, CVMDiskGroup and CVMVolume type information


Activating CFS in the VCS Environment

This operation does not remove the CVMDiskGroup, CVMVolume, or other resources ifthe resources belong to service groups other than CVM. Remove them before proceeding.If any non-CVM resources exist in the CVM group, this operation will fail. You will not beable to remove the configuration if the CVM service group is online. Take the CVM groupoffline on all nodes before proceeding.

Activating CFS in the VCS EnvironmentYou use the /opt/VRTSvxfs/lib/vcs/vxcfsconfig script to dynamically addCFS-specific types and resources (listed under “Resources and Service Groups for FileSystem Cluster Functionality” on page 55) to a running VCS system. VCS must be startedbefore you can run the vxcfsconfig script. You only need to run vxcfsconfig fromone node in the cluster to configure all nodes.

The vxcfsconfig ScriptThe vxcfsconfig script adds the CFS resource types to the VCS configuration.vxcfsconfig does not require any arguments. The script creates a CFSTypes.cf file inthe directory /etc/VRTSvcs/conf/config, and updates the main.cf file. Afteradding the resource types, vxcfsconfig also creates one instance of the CFSfsckdresource type, names the resource, and puts it into the CVM service group. The CFSfsckdresource ensures that a vxfsckd daemon process is running on all nodes.

The vxcfsunconfig ScriptThe vxcfsunconfig script removes the CFS resource types from the VCS configuration.vxcfsunconfig does not require any arguments. The vxcfsunconfig script alsoremoves the CFSfsckd resource.


Guidelines for Creating Resources

Guidelines for Creating ResourcesBelow are suggestions to follow when creating resources for cluster functionality usingthe VCS Cluster Manager GUI.

Creating a Service Group for a Disk Group◆ Put all CVMDiskGroup resources in the CVM service group.

◆ The service group must have the Parallel attribute set.

◆ The service group must have the AutoFailOver attribute cleared.

◆ It is advisable to set the OnlineRetryLimit to 2. If your requirements are different, referto the VERITAS Cluster Server Agents Developer’s Guide under the /opt/VRTSvcsdcdirectory for more information.

Creating a Resource Instance for a Disk Group◆ The CVMDiskGroup instance must have the localized CVMActivation attribute set to

shared-write (sw) for each node.

◆ The CVMDiskGroup instance must be linked to the CVMCluster resource.

Creating a Resource Instance for a Shared Volume◆ The CVMVolume instance must have its CVMDiskGroup attribute set to the name of

the associated disk group (the VxVM name, not the VCS name).

◆ The CVMVolume instance must have its CVMVolume attribute set to the volumename (the VxVM name, not the VCS name).

◆ The CVMVolume instance must be linked to the associated CVMDiskGroup resource.

◆ The CVMVolume instance should have its Critical attribute cleared, otherwise, losinga volume in the service group causes VCS to fault the entire service group. Refer to theonline VCS guides to determine if that is the effect you want.

Creating a Resource Instance for a Cluster File System Mount◆ The CFSMount instance must have the MountPoint attribute set to the full path name

for the mount.

◆ The CFSMount instance must have the BlockDevice attribute set to the full path nameof the device to be mounted.



◆ The CFSMount instance should have the MountOpt attribute set to the desired mountoptions. Leave this blank if you want the default mount options.

◆ The CFSMount instance must be linked to its associated CVMVolume resource.

◆ The CFSMount instance must be linked to the CFSfsckd resource.

◆ The CFSMount instance should have its Critical attribute cleared, otherwise, losing avolume in the service group causes VCS to fault the entire service group. Refer to theonline VCS guides to determine if that is the effect you want.

The illustration below and the example main.cf file (see “Example main.cf File” onpage 61) shows the resource dependencies.

CFS Agent Service Group and Resource Dependencies

CVMVolume=mydg_volume2_res

CVMVolume=mydg_volume2_mount_res

CVMDiskGroup = my_diskgroup_res

CVMVolume = mydg_volume1_res

CFSMount = mydg_volume1_mount_res

link

link

link

link

CVM

CFSfsckd=vxfsckd

link

link



Example main.cf Fileinclude "types.cf"include "CFSTypes.cf"include "CVMTypes.cf"

cluster VCSCluster2 (UserNames = { admin = "cDRpdxPmHpzS." } Administrators = { admin }

CounterInterval = 5)

system star33system star34system star35system star36

group cvm ( SystemList = { star34, star36 = 0, star35 = 0, star33 = 0 } AutoFailOver = 0 Parallel = 1 AutoStartList = { star34, star36, star35, star33 } )

CFSfsckd vxfsckd ( )CVMCluster cvm_clus ( Critical = 0 CVMClustName = VCSCluster2 CVMNodeAddr = { star34 = "169.254.65.1", star36 = "169.254.65.2", star35 = "169.254.65.3", star33 = "169.254.65.4" } CVMNodeId = { star34 = 0, star36 = 1, star35 = 2,

star33 = 3 } PortConfigd = 4500 PortKmsgd = 4501 CVMTimeout = 200 )



CVMMultiNIC cvmmnic ( Critical = 0 CVMDevice @star34 = { qfe0 = "169.254.65.101", qfe1 = "169.254.65.102" } CVMDevice @star36 = { qfe0 = "169.254.65.103", qfe1 = "169.254.65.104" } CVMDevice @star35 = { qfe0 = "169.254.65.105", qfe1 = "169.254.65.106" } CVMDevice @star33 = { qfe0 = "169.254.65.107", qfe1 = "169.254.65.108" } CVMAddress = { star34 = "169.254.65.1",

star36 = "169.254.65.2", star35 = "169.254.65.3", star33 = "169.254.65.4" } ActivePvtNet = 1 CVMNetMask = "0xffffff00" CVMIfconfigTwice = 1 CVMArpDelay = 5 )

cvm_clus requires cvmmnic

// resource dependency tree // // group cvm // { // CVMCluster cvm_clus // { // CVMMultiNIC cvmmnic // } // CFSfsckd vxfsckd // }



Example CVMTypes.cf Filetype CVMCluster (static int OnlineRetryLimit = 2static int OnlineTimeout = 400static str ArgList[] = { CVMClustName, CVMNodeAddr, CVMNodeId,

PortConfigd, PortKmsgd, CVMTimeout }NameRule = ""str CVMClustNamestr CVMNodeAddr{}str CVMNodeId{}int PortConfigdint PortKmsgdint CVMTimeout

)

type CVMDiskGroup (static keylist RegList = { CVMActivation }static str ArgList[] = { CVMDiskGroup, CVMActivation }NameRule = ""str CVMDiskGroupstr CVMActivation

)

type CVMMultiNIC (static keylist RegList = { ActivePvtNet }static int MonitorTimeout = 120static int RestartLimit = 100static str ArgList[] = { CVMDevice, CVMAddress, ActivePvtNet,

CVMNetMask, CVMIfconfigTwice, CVMArpDelay, CVMRouteOptions }NameRule = ""str CVMDevice{}str CVMAddress{}int ActivePvtNetstr CVMNetMaskint CVMIfconfigTwiceint CVMArpDelay = 1str CVMRouteOptions

)

type CVMVolume (static str ArgList[] = { CVMVolume, CVMDiskGroup }NameRule = resource.CVMDiskGroup + "_" + resource.CVMVolumestr CVMVolumestr CVMDiskGroup

)



Example CFSTypes.cf Filetype CFSMount (static int RestartLimit = 2static str ArgList[] = { MountPoint, BlockDevice, MountOpt }NameRule = resource.MountPointstr MountPointstr BlockDevicestr MountOpt

)

type CFSfsckd (static int RestartLimit = 2static str ArgList[] = { }NameRule = ""

)


CFSMount Agent

CFSMount Agent

Description Brings online, takes offline, and monitors a cluster file system mount point. TheCFSMount agent executable is /opt/VRTSvcs/bin/CFSMount/CFSMountAgent.The type definition is in the file /etc/VRTSvcs/conf/config/CFSTypes.cf. Youmust manually add the configuration you want to the main.cf file.

Entry Points - Online—Mounts a block device in cluster mode.

- Offline—Unmounts the file system, forced unmount if necessary, sets primary tosecondary if necessary.

- Monitor—Determines if the file system is mounted. Checks mount status usingthe fsclustadm command.

- Clean—A null operation for a cluster file system mount.

Required Attributes Type andDimension

Definition

BlockDevice string-scalar Block device for mount point.

MountPoint string-scalar Directory for mount point.

Optional Attributes Type andDimension

Definition

MountOpt string-scalar Options for the mount command. To create a valid MountOptattribute string:

- Use VxFS type-specific options only.

- Do not use the –o flag to specify the VxFS-specific options.

- Do not use the –F vxfs file system type option.

- The cluster option is not required.

- Specify options in a comma-separated list as in these examples:

roro,clusterblkclear,mincache=closesync


CFSMount Agent

Type Definitiontype CFSMount (static int RestartLimit = 2static str LogLevelstatic str ArgList[] = {MountPoint, BlockDevice, MountOpt}NameRule = resource.MountPointstr MountPointstr BlockDevicestr MountOpt

)

Sample ConfigurationCFSMount testdg_test01_fsetpri (Critical = 0mountPoint = “/mnt1”BlockDevice = “/dev/vx/dsk/testdg/test01”)

CFSMount testdg_test02_fsetpri (Critical = 0MountPoint = “/mnt2”BlockDevice = “/dev/vx/dsk/testdg/test02”MountOpt = “blkclear,mincache=closesync”)


CFSfsckd Agent

CFSfsckd Agent

Type Definitiontype CFSfsckd (static int RestartLimit = 2static str LogLevelstatic str ArgList[] = { }NameRule = ““

)

Sample ConfigurationCFSfsckd vxfsckd ()

Description Starts, stops, and monitors the vxfsckd process. The CFSfsckd agent executable is/opt/VRTSvcs/bin/CFSfsckd/CFSfsckdAgent. The type definition is in the file/etc/VRTSvcs/conf/config/CFSTypes.cf. The configuration is added to themain.cf file after running the vxcfsconfig script (see “Running the CFS AgentConfiguration Script” on page 34).

Entry Points - Online—Starts the vxfsckd process.

- Offline—Stops the vxfsckd process.

- Monitor—Checks whether the vxfsckd process is running.


Definition

None


Definition

None


CVMCluster Agent

CVMCluster Agent

Type Definitiontype CVMCluster (static int OnlineRetryLimit = 2static int OnlineTimeout = 400static str ArgList[] = { CVMClustName, CVMNodeAddr, CVMNodeId,

PortConfigd, PortKmsgd, CVMTimeout }NameRule = ""str CVMClustNamestr CVMNodeAddr{}str CVMNodeId{}int PortConfigdint PortKmsgdint CVMTimeout

)

Description Controls node membership on the cluster port associated with CVM. TheCVMCluster resource requires the CVMMultiNIC resource and must be configured todepend on CVMMultiNIC. The CVMCluster agent executable is/opt/VRTSvcs/bin/CVMCluster/CVMClusterAgent. The type definition is inthe file /etc/VRTSvcs/conf/config/CVMTypes.cf. The configuration is addedto the main.cf file after running the vxcvmconfig script (see “Running the CVMAgent Configuration Script” on page 31).

Entry Points - Online—Joins a node to the CVM cluster port.

- Offline—Removes a node from the CVM cluster port.

- Monitor—Monitors the node’s CVM cluster membership state.


Definition

CVMClustName string-scalar Name of the cluster.

CVMNodeAddr string-association List of host names and IP addresses.

CVMNodeId string-association List of host names and LLT node numbers.

PortConfigd integer-scalar Port number used by CVM for vxconfigd-level communication.

PortKmsgd integer-scalar Port number used by CVM for kernel-level communication.

CVMTimeout integer-scalar Timeout used by CVM during cluster reconfigurations.


CVMCluster Agent

Sample ConfigurationCVMCluster cvm_clus (

Critical = 0CVMClustName = vcsCVMNodeAddr = { node1 = "xxx.254.65.1", node2 = "xxx.254.65.2" }CVMNodeId = { node1 = 1, node2 = 2 }PortConfigd = 4500PortKmsgd = 4501CVMTimeout = 200

)


CVMDiskGroup Agent

CVMDiskGroup Agent

Type Definitiontype CVMDiskGroup (static keylist RegList = { CVMActivation }static str ArgList[] = { CVMDiskGroup, CVMActivation }NameRule = ""str CVMDiskGroupstr CVMActivation

)

Sample ConfigurationCVMDiskGroup testdg (

CVMDiskGroup = testdgCVMActivation @node1 = swCVMActivation @node2 = sw)

Description Brings online, takes offline, and monitors a VxVM shared diskgroup. TheCVMDiskGroup resource requires the CVMCluster resource. The CVMDiskGroupagent executable is /opt/VRTSvcs/bin/CVMDiskGroup/CVMDiskGroupAgent.The type definition is in the file /etc/VRTSvcs/conf/config/CVMTypes.cf. Youmust manually add the configuration you want to the main.cf file.

Entry Points - Online—Imports the diskgroup in shared mode if it did not auto-import.

- Offline—Records the shared diskgroup as offline.

- Monitor—Determines whether the diskgroup is online.


Definition

CVMDiskGroup string-scalar Shared diskgroup name.

CVMActivation string-scalar Activation mode for the diskgroup. Must be set to shared-write(sw). This is a localized attribute.


CVMMultiNIC Agent

CVMMultiNIC Agent

Description Represents a set of network interfaces on private networks, and fails over the virtualIP address used by CVM between the interfaces. Each interface has a base IP address,and the agent configures them if not already configured. The set of interfaces mustcontain only one interface per private network. Their specified order is the order inwhich the corresponding private networks are selected during a failover as the targetfor migrating the virtual IP address used by CVM. The private network correspondingto the first interface entered is the primary network. The interface on which CVM’svirtual IP address is currently configured is the active interface.

The CVMMultiNIC agent executable is/opt/VRTSvcs/bin/CVMMultiNIC/CVMMultiNICAgent. The type definition is inthe file /etc/VRTSvcs/conf/config/CVMTypes.cf. The configuration is addedto the main.cf file after running the vxcvmconfig script (see “Running the CVMAgent Configuration Script” on page 31).

Entry Points - Online—Configures the base IP address on the interfaces if they are not alreadyconfigured. Configures the virtual IP address used by CVM on the interfacepointed to by the ActivePvtNet attribute.

- Offline—Removes the virtual IP address from the active interface, which is pointedto by the ActivePvtNet attribute.

- Monitor—Checks for activity on the currently active interface. If it does not detectactivity, it initiates the failover of virtual IP address used by CVM to the next listedinterface. It modifies the value of the ActivePvtNet attribute to point to the failedover interface. This entry point also checks if the currently active interface is thesame as the one pointed to by the ActivePvtNet attribute. If it is not the same, itmigrates the virtual IP address to the one pointed to by ActivePvtNet. Thishappens when another system in the cluster initiated a failover and updatedActivePvtNet to the new value.


Definition

CVMDevice string-association List of interfaces and their base IP addresses. The interface order isthe order in which the corresponding private networks areselected for migrating the virtual IP address used by CVM, so thisorder must be the same on all nodes. This is a localized attribute.

CVMAddress string-association List of nodes and their virtual IP addresses.

ActivePvtNet integer-scalar Points to the interface on which the virtual IP address is currentlyconfigured.

CVMNetMask string-scalar Netmask for the IP addresses.


CVMMultiNIC Agent

Type Definitiontype CVMMultiNIC (static keylist RegList = { ActivePvtNet }static int MonitorTimeout = 120static int RestartLimit = 100static str ArgList[] = { CVMDevice, CVMAddress, ActivePvtNet,

CVMNetMask, CVMIfconfigTwice, CVMArpDelay, CVMRouteOptions }NameRule = ""str CVMDevice{}str CVMAddress{}int ActivePvtNetstr CVMNetMaskint CVMIfconfigTwiceint CVMArpDelay = 1str CVMRouteOptions

)

Sample ConfigurationCVMMultiNIC cvmmnic (

Critical = 0CVMDevice @node1 = { qfe0 = "xxx.254.65.101", qfe3 = "xxx.254.65.102" }

CVMDevice @node2 = { qfe0 = "xxx.254.65.103", qfe3 = "xxx.254.65.104" }

CVMAddress = { node1 = "xxx.254.65.1", node2 = "xxx.254.65.2" }ActivePvtNet = 1CVMNetMask = "0xffffff00"CVMIfconfigTwice = 1CVMArpDelay = 5


Definition

CVMIfconfigTwice integer-scalar Configures an IP address twice.

CVMArpDelay integer-scalar Number of seconds to sleep between configuring an interface andsending out a broadcast to notify routers about base IP address.

CVMRouteOptions string-scalar String to add a route when configuring an interface.


CVMVolume Agent

CVMVolume Agent

Type Definitiontype CVMVolume (static str ArgList[] = { CVMVolume, CVMDiskGroup }NameRule = resource.CVMDiskGroup + "_" + resource.CVMVolumestr CVMVolumestr CVMDiskGroup

)

Sample ConfigurationCVMVolume testdg_test01 (

Critical = 0CVMVolume = test01CVMDiskGroup = testdg)

Description Brings online, takes offline, and monitors a VxVM volume belonging to a shareddiskgroup. The CVMVolume resource requires the corresponding CVMDiskGroup.The CVMVolume agent executable is/opt/VRTSvcs/bin/CVMVolume/CVMVolumeAgent. The type definition is inthe file /etc/VRTSvcs/conf/config/CVMTypes.cf. You must manually add theconfiguration you want to the main.cf file.

Entry Points - Online—Records the volume as being online.

- Offline—Records the volume as being offline.

- Monitor—Determines whether the volume is online.


Definition

CVMDiskGroup string-scalar Name of the shared diskgroup, to which the volume belongs.

CVMVolume string-scalar Name of the volume.


CVMVolume Agent


Hardware Configuration
B VERITAS cluster file systems operate best in a “homogenous” environment, that is, usingcomparable hardware, software, and shared storage configurations. Because CFS tries todistribute computation and memory across different cluster nodes, each node is expectedto have the same power and performance capabilities. Lesser quality systems, forexample, those with less RAM or fewer processors, can become a performance bottleneck,even if there is not much load on it at the application level. Worse, systems may not beable to adequately handle failover situations.
In Fibre Channel configurations, Fibre Channel switches and disk arrays must be capableof operating in switched mode. Also, the Fibre Channel fabric and disk arrays comprisingthe cluster must be capable of handling cluster membership changes caused by rebootsand power cycling.

Fibre Channel Switch Example ConfigurationOne possible backbone of a Fibre Channel fabric is a Brocade switch. If you are using aswitch to connect your disk storage to the cluster, you must configure the switch for useon Ethernet by providing the standard TCP/IP configuration parameters—IP address,netmask, and default router. The instruction guides for the switch describe how to do this.There are two ways to configure Brocade switches, through a serial port (SilkWorm 2400)or through the front panel (SilkWorm 2800).

1. After the Fibre Channel switch is configured, open a telnet session to the switchand log in as admin with the password password.

> telnet switch_name

login: adminPassword: password

75

Cluster Platform Setup

2. Check the status of the switch and whether it sees the SAN-attached storage:

switch:admin> switchShow

port 0: sw Online F-Port 20:00:00:e0:69:c0:11:d2port 1: sw Online F-Port 20:00:00:e0:69:c0:11:e6port 2: sw Online F-Port 20:00:00:e0:69:c0:12:09port 3: sw Online F-Port 20:00:00:e0:69:c0:12:7dport 4: sw Online L-Port 1 publicport 5: sw No_Light L-Portport 6: sw No_Light L-Portport 7: sw No_Light L-Portvalue = 8 = 0x8


1. Determine whether your Sun platform uses the PCI or SBus architecture and installthe appropriate JNI Fibre Channel SCSI/IP host bus adapter (HBA) card (see theSANPoint Foundation Suite HA Release Notes).

2. Connect the platform to the Fibre Channel switch or disk array with a fiber optic cable(see the figure under “Hardware Overview” on page 2).

3. Power up the platform and install the Solaris OS.

4. Mount the JNI CD (if you do not have the CD, you can obtain the latest drivers at theJNI Web site: http://www.jni.com/drivers) and add the corresponding JNIFibre Channel software package (fcaw.pkg for SBus, fca-pca.pkg for PCI bus):

# /usr/sbin/pkgadd -d fca-pci.pkg

Specify All when prompted for which packages to process.

5. Edit the driver configuration file (/kernel/drv/fcaw.conf for SBus,/kernel/drv/fca.pci.conf for PCI). Set the value of fca_nport to 1:

fca_nport=1



6. Reboot the system. During the reboot, messages from the Fibre Channel HBA displayon the terminal. Here is some sample output from a PCI system after a successfulinitialization:

JNI Fibre Channel Adapter model FCI-1063fca-pci0: SCSI ID 125 / AL_PA 0x1fca-pci0: Fibre Channel WWNN: 100000e069c011d2 WWPN:200000e069c011d2fca-pci0: FCA SCSI/IP Driver Version 2.7, Aug 19, 2001 for

Solaris 8fca-pci0: All Rights Reserved.NOTICE: pci1242,46430 NPORT Initialization Complete, SID=c011d2NOTICE: pci1242,46430 NPORT Initialization Complete, SID=c011d2

7. After the HBA driver is installed, check whether the shared disks are accessible. Runthe format command on all nodes:

# formatAVAILABLE DISK SELECTIONS:0. c0t0d0 <IBM-DJNA-370910 cyl 17660 alt 2 hd 16 sec 63>

/pci@1f,0/pci@1,1/ide@3/dad@0,01. c1t0d0 <VERITAS--0001 cyl 25598 alt 2 hd 64 sec 64>

/pci@1f,0/pci@1/fibre-channel@1/sd@0,0

Appendix B, Hardware Configuration 77

Index

Aactivating resources

CFS 58CVM 57

adding a new node to a cluster 37agents

CFS 53CFSfsckd 67CFSMount 65CVM 53CVMCluster 68CVMDiskGroup 70CVMMultiNIC 71CVMVolume 73modifying 55VCS 53VCS bundled 53

attributes defined 54

CCFS

activating resources 58agents 53deinstalling 43installing using the cfsinstall utility 25packages 3

CFSfsckd agent 67cfsinstall command 25CFSMount agent 65CFSTypes.cf file example 64checking for mounted file systems 43cluster

adding a new node 37invalid name 24naming requirement 8platform setup 76upgrading 48verifying operation 20

ClusterService Group configuration 11

command failures 41commands

cfsinstall 25df 43format 77gabconfig 15, 19, 47hastart 24hastatus 20hastop 47licensevcs 6lltconfig 14, 47lltstat 17mkfs 36modunload 47mount 36pkgadd 28pkgrm 44, 47switchShow 76telnet 75vxassist 36vxclustadm 34vxdctl 34vxinstall 30vxlicense 26vxplex 44vxprint 30, 36

configuration filesgabtab 15llthosts 14llttab 14main.cf 16, 61modifying 55types.cf 16, 54

configuringFibre Channel switch 75VxVM 30

creating resource and service groups 59customer support x

79

CVMactivating resources 57agents 53

CVMcluster agent 68CVMDiskGroup agent 70CVMMultiNIC agent 71CVMTypes.cf file example 63CVMVolume agent 73

Ddeinstalling

CFS 43VCS 46

dependencies of resource and servicegroups 56df command, using to check for mountedfile systems 43

Eemail notification (SMTP) 12encapsulating rootdg 30environment variables 4

FFibre Channel

switch configuration 75using with CFS 2

file systemsCFSMount agent monitoring 65making and mounting 36

format command 77

GGAB

port membership information 19verifying 19

gabconfig command 19-a (verifying GAB) 19in gabtab file 15specify seeding 15-U (stop GAB) 47

gabtab file 15

Hhardware

example setup 2, 76hastart command 24hastatus -summary command 20hastop command 47help, customer support x

Iinstallation 12

cfsinstall utility 25installation and configuration scripts 4installvcs utility 6license keys 26verifying GAB 19verifying LLT 14verifying VCS 14

installvcs utility 7

Llicense keys, installing 26licensevcs command 6links, private network 14LLT, verifying 17lltconfig command 14, 47llthosts file 14lltstat command 17llttab file 14

Mmain.cf file

created 15, 49example 61

managing resource and service groups 55MANPATH environment variable 4manual pages

potential problems 41troubleshooting 41

membership information 19mkfs command 36modifying agents 55modunload command 47mount command

mounting in shared mode 36potential problems 42

Ppackages

CFS 3VCS 3

PATH environment variable 4pkgadd command 28pkgrm command 44, 47port membership information 19problems

accessing manual pages 41executing file system commands 41mounting and unmounting file systems


42

Rresource and service groups

creating 59defined 54dependencies 56managing 55

rootdg, encapsulating 30

SSAN, see Storage Area Networkscripts

installation and configuration 4vxcfsconfig 30, 34, 58vxcvmconfig 30, 31, 57

service groups 54setup, cluster platforms 76starting VCS during installvcs script 13Storage Area Network 2switchShow command 76system state attribute value 20

Ttechnical support xtelnet command 75troubleshooting

accessing manual pages 41executing file system commands 41mounting and unmounting file systems42

types.cf file 16, 54

Uuninstallvcs utility 46upgrading a cluster 48

VVCS

attributes 54bundled agents 53configuration files

main.cf 16, 61types.cf 16

deinstalling 46documentation 27installing using installvcs utility 6packages 3starting 13

vxassist command 36vxcfsconfig script

description 58installing CFS agent 30running 34

vxclustadm command 34vxcvmconfig script

description 57installing CVM agent 30running 31

vxdctl command 34vxinstall command 30vxlicense command 26vxplex command 44vxprint command 30, 36VxVM configuration 30

WWeb Console

accessing after installation 27configured in ClusterService Group 11

Index 81

solaris - giovanni mantengoli cluster file system 3... · command failures ... list of cfs agents...

Documents