Dell C

TeciSCS 

Confidential 

chnical ISI suppor

nstructiort for IBM

on M AIX 

For Authhorized Use O

 

Only

Documen

Revision: Revision D

nt ID: N/A 

A02 Date: Septemmber 2, 2009

 

iSCSI support for IBM AIX

Disclaimer: This information is provided only for support purposes. EqualLogic does not

develop or maintain iSCSI initiators. Check with your initiator vendor for the latest information.

To connect to PS Series iSCSI volumes from an AIX host, you must be running PS Series

firmware version 2.2.3 or later. For versions of AIX from 5.1 up to 5.2 ML2, the Cisco software

initiator is downloadable from the EqualLogic support web site. For AIX 5.2 ML7, and AIX 5.3

ML3 and higher, IBM is including an iSCSI initiator in the operating system itself.

IBM also makes an HBA card (FC 5714, 1987, 5713, 1986) which can be installed in AIX

systems, based on the QLogic QLA4050 chipset. EqualLogic has not fully tested and qualified

this adapter, and cannot make any guarantees about its specific compatibility with EqualLogic

storage arrays. Please see your IBM reseller for specific compatibility requirements, and review

the following link: IBM 1 Gigabit iSCSI TOE Adapter Installation and Configuration.

AIX 5.1 and 5.2 up to ML2 (32-bit only):

Use the Cisco iSCSI initiator 3.4.5 for IBM AIX (“AIX 5.1/5.2 iSCSI Initiator,” posted

12/08/2004) available from the EqualLogic Support web site. Use is subject to license agreement

acceptance. Be sure to review the initiator release notes for package dependency information,

and the related Knowledge Base article on the EqualLogic Customer Support site titled Using the

AIX iSCSI Initiator. Please also read the README file included in the initiator TAR file, as it

contains detailed information about installing and using the initiator, including dependencies. If

you need to report an issue, contact EqualLogic Customer Support.

AIX 5.2 ML7 (32- and 64-bit):

AIX iSCSI support is now part of the operating system distribution. This requires ML7 and

U803038. When you have installed maintenance level 7, go to Quick Links for AIX Fixes

(http://www.ibm.com/servers/eserver/support/unixservers/aixfixes.html). Under Specific Fixes,

click AIX 5.2 and enter U803038 in the search string field to obtain the correct fix.

AIX 5.3 (32- and 64-bit):

AIX iSCSI support is now part of the operating system distribution. This requires ML3 and

U802416. When you have installed maintenance level 3, go to Quick Links for AIX Fixes

(http://www.ibm.com/servers/eserver/support/unixservers/aixfixes.html). Under Specific Fixes,

click AIX 5.3 and enter U802416 in the search string field to obtain the correct fix.

IBM directly supports the native iSCSI initiator in AIX 5.3. On-line documentation for the

package can be found at the AIX Information Center

(http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.doc/aixbma

n/commadmn/tcp_initiator.htm).

Note that there is no support for AIX 5.2 ML3-ML6. If you need to use AIX 5.2 later than ML2,

please update to ML7 and use the initiator that is included with the operating system.

Configuration Considerations

First and foremost, it’s useful to determine that actual speed that your AIX ethernet intreface has

negotiated with the switch. If the interface has negotiated something other than 1000Mbps Full

Duplex, then your performance is already compromised. To determine what speed your ethernet

interfaces are currently using, issue the following command:

# netstat -v | egrep "STATISTICS|Media Speed"

ETHERNET STATISTICS (ent0) :

Media Speed Selected: Auto negotiation

Media Speed Running: 100 Mbps Full Duplex

ETHERNET STATISTICS (ent1) :

Media Speed Selected: Auto negotiation

Media Speed Running: 1000 Mbps Full Duplex

#

Note that on the example system, there is a built-in 10/100 port, and an added 1000Mbps port.

This command tells us that the built-in port has properly negotiated 100Mbps, and that the

gigabit port has properly negotiated 1000Mps. Using “ifconfig” and “netstat -r”, verify the

addressing and routing information to assure yourself that the gigabit port is the one that will

actually be communicating with the array.

To ensure the best performance, tune network options and interface parameters for maximum

iSCSI I/O throughput on the AIX system. Note that Jumbo Frames should only be enabled if

your switch supports them. The remainder of this guide assumes that your switch is

appropriately configured to enable Jumbo Frames, so you should ignore the Jumbo Frames

information if you are not using them.

Enable the TCP Large Send, TCP send and receive flow control, and Jumbo Frame features of

the AIX Gigabit Ethernet Adapter and the iSCSI Target interface (as appropriate). Use the

following commands to see and set those parameters:

# lsattr -E -l ent1

# chdev -l ent1 -a flow_ctrl=yes

# chdev -l ent1 -a large_send=yes

# chdev -l ent1 -a jumbo_frames=yes

If a user or system service (for example, IP interface) is currently using the ethernet device, you

may see a message similar to the following:

Method error (/usr/lib/methods/ucfgcommo):

0514-062 Cannot perform the requested function because the

specified device is busy.

To configure the ethernet device, you would need to first detach it. For example, if the user is IP

interface en1, then you can use the following command:

# ifconfig en1 detach

Then go back and re-run the chdev commands that failed.

There are networking parameters that should be set in SMIT as well for the interface

communicating with the PS Series array. Starting from “smit tcpip” (which is the

“Communications / TCP/IP” menu): select “Further Configuration / Network

Interfaces / Network Interface Selection / Change”, select the appropriate interface

(en1 in the current example), and make the following settings:

Interface Specific Network Options

('NULL' will unset the option)

rfc1323 [1]

tcp_mssdflt []

tcp_nodelay []

tcp_recvspace [262144]

tcp_sendspace [262144]

Apply the changes, and go back to the “Network Interfaces / Network Interface

Drivers” menu. Select the appropriate interface, and set the “Maximum IP PACKET SIZE” to

9000.

The iSCSI Software Initiator's maximum transfer size is 256KB. Assuming that the system

maximums for tcp_sendspace and tcp_recvspace are set to 262144 bytes, an ifconfig

command used to configure a gigabit Ethernet interface might look like the following:

ifconfig en1 10.1.2.216 mtu 9000 tcp_sendspace 262144 tcp_recvspace 262144

Set the sb_max network option to at least 524288, and preferably 1048576. Use the following

commands to show the current values, and to set sb_max to 1048576:

# no -a

# no -o sb_max

# no -p -o sb_max=1048576

Important Note: If you configure Jumbo Frames on an AIX host, you may find that large data

transfers can freeze your system for minutes at a time. The problem is that the way AIX

implements the “Nagle algorithm” for aggregating small packets, automatically applies to every

packet. With a 1500-byte MTU, this is not a serious impact. But when the packet size goes to

9000 bytes, it can take a long time to fill up the packet, and more often the 200ms Nagle timeout

is hit before the packet is transmitted.

AIX has a tunable parameter to set the threshhold for what is a "large" packet and what is a

"small" packet to the Nagle algorithm. When set to 1900 bytes, it forces the typical 2KB iSCSI

data segment to be considered a “large” packet and so it is sent immediately.

The AIX command to set the Nagle limit to 1900 is:

# no -o tcp_nagle_limit=1900

This is a system-wide parameter.

Configuring Etherchannel Failover on AIX

Under AIX, it is possible to configure a pair of ethernet adapters to provide failover protection.

The mechanism is called "Etherchannel", and the general process is to create a new virtual

etherchannel adapter that is composed of two physical ethernet adapters. Currently, only fail-

over is supported.

The general process for creating and configuring a new Etherchannel device is to identify the

ethernet ports you wish to use, detach them (remove the devices), create the etherchannel device,

assign the primary interface, and assign the backup interface. Then configure the new

etherchannel device as you would a normal ethernet device.

1. Make sure the Ethernet adapters used for Etherchannel all in “detach” state (unconfigured).

For example, the commands below will unconfigure en0 and en1.

# ifconfig en0 down detach

# ifconfig en1 down detach

# rmdev -l en0

# rmdev -l en1

# rmdev -dl ent0

# rmdev -dl ent1

2. Change “Enable Link Polling” value to "yes" on any 100 Mbps interfaces to be included in

the channel:

# smit chgenet

Select the appropriate ethernet interface (entX) device. Set "Enable Link Polling" for the

device to "yes" and activate the change (hit Return).

3. Create an etherchannel:

# smit etherchannel

Select "Add an Etherchannel", and then select your primary interface. This will be the

intreface preferentially used. Select only one interface.

Select "Backup Adapter" by moving the cursor to the correct line and hitting PF4 (or ESC-4),

and then selecting another ethernet interface. This interface will only be used when the main

interface is not operational. Set "Enable Gigabit Ethernet Jumbo Frames" to "yes" if

applicable to your network. Leave the other settings as default.

Press Enter to execute. The system will reply with the newly created ent device name: "ent2

Available".

4. Assign IP to newly created EtherChannel interface:

# smit chinet

Select the new entX device, and configure it as you would have your original primary address.

In general, make sure the EtherChannel Ports on the switch are on the same segment. Special

note for the Dell 5224: avoid Port 21, 22, 23 and 24.

Limitations

Cisco initiator with AIX 5.1 and 5.2 up to ML2:

Refer to the driver release notes and README for installation information.

AIX imposes the following restrictions on storage devices:

AIX does not support dynamic expansion of physical volumes.

To increase the size of the volume group, create another volume on the array group, and

add the new volume to the AIX VG (see Extending a Volume Group).

The interface used for iSCSI traffic must be set to full duplex.

The initiator will automatically discover and login to any visible iSCSI targets on the

array(s) it is configured to use. You should use IP address-based access control on the

storage array to ensure that the initiator does not access any incorrect volumes.

AIX native initiator AIX 5.2 ML7 and 5.3 ML3 and above:

The iSCSI software initiator does not implement iSCSI inbound discovery (For example,

Canonical iSCSI targets). A text file is used to configure each target.

IPv6 is not implemented.

Only CHAP(MD5) can be used to configure Initiator authentication. Target

authentication is not implemented.

The default queue_depth for iSCSI hdisk devices is 1. This should be changed to 128.

To avoid configuration problems and error log entries when you create volume groups

using iSCSI devices, follow these guidelines:

o Configure volume groups that are created using iSCSI devices to be in an inactive

state after reboot. After the iSCSI devices are configured, manually activate the

iSCSI-backed volume groups. Then, mount any associated file systems.

o Volume groups are activated during a different boot phase than the iSCSI

software driver. For this reason, it is not possible to activate iSCSI volume groups

during the boot process. It may be possible to write a custom startup script to

perform these operations later in the boot process (see Startup/Shutdown Scripts

below).

o Do not span volume groups across non-iSCSI devices.

If connectivity to iSCSI target devices is lost, I/O failures occur. To prevent I/O failures

and file system corruption, stop all I/O activity and unmount iSCSI backed file systems

before doing anything that will cause long term loss of connectivity to active iSCSI

targets.

If a loss of connectivity to iSCSI targets occurs while applications are attempting I/O

activities with iSCSI devices, I/O errors will eventually occur. It might not be possible to

unmount iSCSI backed file systems because the underlying iSCSI device stays busy.

Administrator must manually unmount all iSCSI filesystems, and varyoffvg all iSCSI

volume groups before rebooting or shutting down, otherwise the server may hang during

shutdown when it tries to unmount a filesystem after the iSCSI system has stopped (see

Startup/Shutdown Scripts below).

IBM 1 Gigabit iSCSI TOE Adapter::

The default queue_depth for iSCSI hdisk devices is 1. This should be changed to 128.

The "Maximum number of commands to queue to the adapter" parameter for the HBA

ports should be set to be 128 times the number of volumes connected.

MPIO is not supported between the IBM HBA cards and EqualLogic PS Series storage

arrays. If you have multiple HBA ports, you should split your volumes among your ports,

but be careful to only connect once to any volume.

Security considerations

AIX 5.2 ML7 and 5.3 ML3 and above:

The /etc/iscsi/directory file and the /etc/iscsi/targets configuration file are protected from non-

privileged users through file permission and ownership. CHAP secrets are saved in the

/etc/iscsi/targets file as clear text.

Note: Do not to change the original file permission and ownership of these files.

Using the IBM AIX native iSCSI initiator Under AIX 5.2 ML7 and 5.3 ML3 and above

After installing the required maintenance levels and any required fixes, and configuring the

network settings appropriately, you are ready to configure your iSCSI targets. The AIX native

iSCSI controller does not perform device discovery, so you will need to know your iSCSI IQN

target names specifically.

Verify the installation by going into SMIT (also see the SMIT section below) and verify that

there is an iSCSI heading under Devices. Select iSCSI, and iSCSI Protocol Device, and then

Change / Show Characteristics. You may desire to change the iSCSI Initiator Name that was

assigned by default, to better match your internal naming conventions. Please remember that a

legal iSCSI name must begin with “iqn.” or “eul.” To make sure that any name you assign here is

legal.

The first step to getting iSCSI storage on your host is to create an iSCSI volume on your PS

Series array group. On general principle, you would normally assign an IP address-based access

control list to this volume at this time, so that only your desired AIX host can use it. However,

for an initial configuration and verification, leave the volume configured for open access. Once

communication has been verified, then enable the ACL. If you also wish to configure CHAP

authentication, see Limitations above.

To enable and use this iSCSI volume as an hdisk device under AIX, you must add it to the iSCSI

targets file, and reconfigure your iSCSI device. Once the device is visible as an hdisk, then you

can operate on it just as you would a local physical disk (except for boot-time - see Limitations

and Startup/Shutdown Scripts), including creating a new volume group, logical volume, and

filesystem.

Note that the AIX native initiator does not keep connections to the PS Series array group active

when there is no data being transferred. A new volume that has been attached using cfgmgr is

not necessarily going to show an iSCSI connection in the PS Array GUI at any given moment.

When the volume is attached to a volume group and has a filesystem placed on it, then AIX will

maintain the iSCSI connection.

SMIT

Before attempting to configure the iSCSI initiator under AIX, first verify that it is enabled and

operational. The right place to do this is within SMIT. Run "smit iscsi" from a shell prompt,

and select the iSCSI Protocol Device. First, "List All iSCSI Protocol Devices", and verify that

you see this output:

iscsi0 Available iSCSI Protocol Device

Then hit F3 and select "Change / Show Characteristics of an iSCSI Protocol Device". Select

"iscsi0", and verify that you have something like the folllowing output:

Change / Show Characteristics of an iSCSI Protocol Device

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

iSCSI Protocol Device iscsi0

Description iSCSI Protocol Device

Status Available

iSCSI Initiator Name [iqn.localhost.hostid.>

Maximum number of commands to queue to driver [384] +#

Discovery Policy file +

Maximum Targets Allowed [16] +#

Apply change to DATABASE only no +

F1=Help F2=Refresh F3=Cancel F4=List

F5=Reset F6=Command F7=Edit F8=Image

F9=Shell F10=Exit Enter=Do

The important thing to look for is the Discovery Policy, which must be "file" (and not "ODM").

The other settings are fair to leave at the default. The "Maximum number of commands to queue

to driver" is set elsewhere in this guide, the "iSCSI Initiator Name" can be edited as long as the

resulting name is legal, and begins with "iqn.". You may need to adjust the "Maximum Targtes

Allowed" entry to suit your specific needs.

/etc/iscsi/targets

The iSCSI targets file lives at /etc/iscsi/targets, and defines every volume that the AIX host will

attempt to connect to. The format of the file is that each line defines a single target (line breaks

can be escaped) with the following fields:

<IP address of the PS Array> <Port> <IQN name> [“plaintext CHAP secret”]

A sample line might look like this:

172.19.0.100 3260 iqn.2001-05.com.equallogic:6-...-volume-name

You should always use the group address for the PS Series array group, and not any of the

individual eth port addresses. This allows the array to balance its network load normally. The

port will almost always be 3260, which is the standard iSCSI service port number. It is

necessary to know the IQN name for the volume on the PS Array; this information is displayed

in the volume configuration in the array GUI and CLI, and can be copied and pasted as necessary.

Create a line for the target you just created.

Adding drives using cfgmgr

The command to process any changes to the iSCSI configuration and add any new devices is

cfgmgr. To limit the scope to just iSCSI volumes, enter the following command:

# cfgmgr -l iscsi0

This will take a few seconds to operate, and then any new targets that were added since the last

time the command was run will be assigned new hdisk devices. To view all of your hdisk

devices, enter the following command:

# lsdev -C -c disk

hdisk0 Available 1S-08-00-8,0 16 Bit LVD SCSI Disk Drive

hdisk1 Available 1S-08-00-3,0 16 Bit LVD SCSI Disk Drive

hdisk2 Available Other iSCSI Disk Drive

#

In this example, the AIX host has two local SCSI devices, and one PS Array volume attached.

hdisk2 is available to be used as a new volume group or as an addition to an existing VG. Note

the limitation stated above: never mix iSCSI and non-iSCSI devices in a single volume group.

Corruption will result.

To determine what iSCSI target a particular hdisk device is connected to, use the following

command, and look for the following two device attributes:

# lsattr -E -l hdisk2

host_addr 172.19.0.100

target_name iqn.2001-05.com.equallogic:6-...-volume-name

These two parameters tell you the IP address that the iSCSI target was logged in through, and the

IQN name of the iSCSI target. It is an EqualLogic convention to place the name of the volume

at the end of the IQN name for every target.

Once the new hdisk device has been identified, you can modify the device properties (see Queue

Depth below), go into SMIT and configure it as a new volume group, and create an LV and a

filesystem on it, or you can manipulate the device from the shell prompt. Managing the AIX

LVM is beyond the scope of this document.

Queue Depth

According to IBM’s documentation on their iSCSI initiator, iSCSI-based disks may benefit from

increasing the queue depth on the drive if the storage array supports the necessary feature sets

(“command tag queueing” and “NACA=1”). The EqualLogic PS Array supports those features

properly, and recommends a queue depth of 128.

Before connecting to your iSCSI volumes, you will need to determine what your queue depth

setting will be for each your connected volumes, so you can set the initiator itself to have a

capacity large enough to service all of the volumes. On the iscsi0 device, there is a "number of

command elements" attribute, which effectively defines the total queue depth for the entire

iscsi0 device tree. The default value is 200. For best performance, this parameter should be

greater than or equal to the sum of all iSCSI hdisk queue_depth settings. For example, if you

have 20 iSCSI devices each with queue_depth=12, the num_cmd_elems parameter should be at

least 240. If it is less than 240 it should still work, though there will be times when the driver will

stall a command waiting for a free command element.

This can be changed at the iSCSI page in smit (run “smitty iscsi” to get to that page), or it can

be set at the command line:

# chdev -l iscsi0 -a num_cmd_elems=384

Note that this parameter can only be changed on the iSCSI initiator when no targets are currently

connected. If you see the following error, either from the command line or in smit, then you will

need to comment out all of your targets in /etc/iscsi/targets and re-run "cfgmgr -l

iscsi0" to clear the connections before you can set it.

Method error (/usr/lib/methods/chggen):

0514-029 Cannot perform the requested function because a

child device of the specified device is not in a correct state.

To change the queue depth for an iSCSI disk device, first confirm the current value, then change

it, then confirm the new value:

# lsattr -E -l hdisk2

queue_depth 1

# chdev -l hdisk2 -a queue_depth=128

hdisk23 changed

# lsattr -E -l hdisk2

queue_depth 128

The change can only be made on a device which is not currently being used – either the device

has not been assigned to a VG, or the VG that it belongs to has been deactivated. If you see this

error:

# chdev -l hdisk2 -a queue_depth=128

Method error (/usr/lib/methods/chgdisk):

0514-062 Cannot perform the requested function because the

specified device is busy.

Then you should unmount any LVs that are on that disk, and go to SMIT and explicitly

deactivate the VG that contains the desired hdisk. Make the change outlined above, and then

reactivate the VG and remount the LVs as necessary.

Note that the hdisk queue_depth parameter cannot be changed within SMIT. SMIT has some

difficulties with iSCSI-based disk parameters, so this change must be done at the command

prompt.

Setting Disk Timeouts

In the event of a short-term network interruption or a controller failover on the PS Array, the

AIX host may stop receiving data from the PS Array for some time. On many operating systems,

it's necessary to configure an extended disk timeout, to have the operating system tolerate much

longer periods of network interruption before failing the I/O operation, or failing the entire disk

device. For every operating system, EqualLogic recommends configuring an operating system to

tolerate up to 120 seconds of communications loss to withstand a full controller failover.

AIX will tolerate at least 120 seconds of communication loss before returning errors to an

application, so it does not require any reconfiguration. Note that an application is still free to

retry its operations even after that period.

There are two places in AIX with parameters which govern how the OS handles timeouts: within

the TCP stack and within the SCSI system. The system-wide tcp_keepidle parameter

determines how long an unclosed TCP session will be kept alive without response. The units for

this are half-seconds, and the system default is 14400 – 2 hours.

The disk-level I/O timeout value is set on each hdisk device separately, and is called

rw_timeout. It defaults to 30 seconds. However, exceeding 30 seconds of I/O timeout only

causes a temporary error to be logged, and no data is lost.

To configure the system-wide tcp_keepidle parameter:

# no -o tcp_keepidle

tcp_keepidle = 14400

# no -o tcp_keepidle=14400

To configure the disk-level I/O timeout parameter:

# lsattr -El hdisk2

rw_timeout 30

# chdev -l hdisk2 -a rw_timeout=120

A temporary I/O error that was recovered appears in the system error log like this (not including

the additional SENSE DATA block):

LABEL: SC_DISK_ERR4

Date/Time: Fri Dec 29 14:35:40 EST 2006

Type: TEMP

Resource Name: hdisk2

Description

DISK OPERATION ERROR

A permanent I/O error that caused an error to be returned to the calling application appears in the

system log like this (not including the additional SENSE DATA block):

LABEL: SC_DISK_ERR3

Date/Time: Fri Dec 29 14:44:20 EST 2006

Type: PERM

Resource Name: hdisk2

Description

DISK OPERATION ERROR

Deleting iSCSI devices

To remove an iSCSI volume from your system, first unmount any applicable filesystems, and

disable the volume group using varyoffvg. You may then also remove the appropriate line from

/etc/iscsi/targets, but note that removing an entry on the iSCSI targets file does not cause AIX to

dismount the volume. It only prevents the AIX host from reattaching to it later.

To actually remove an iSCSI device from your AIX system, you must go to SMIT, to the

Devices / Fixed Disk section, and remove the appropriate volume there. The volume will no

longer be connected to the host until another cfgmgr call is made, but only if the entry in

/etc/iscsi/targets still exists.

Startup/Shutdown Scripts

Since the native AIX initiator does not handle automatically activating iSCSI-based volume

groups or mounting filesystems on them, that process has to be handled manually, or by the use

of a start-up script.

As stated in the Limitations section, you must create your volume groups with the option to not

activate them at boot. Also, you must create your filesystems to not automatically mount at boot.

The iSCSI initiator is not running when the volume group system wants to activate VGs, or when

the automatic mouting of filesystems happens. You must either manually reactivete your VGs

and manually mount your filesystems, or you must create a script to do it for you.

Here is a simple, and not robust, script for connecting an iSCSI volume at startup and

disconnecting it at shutdown. Note that this script does not attempt to detect or handle errors.

The script references a configuration file, by default /etc/iscsi/volumegroups, which should

contain lines declaring each volume group to activate, followed by the mount points for each

filesystem contained within that VG. To use this script, install it as /etc/rc.d/init.d/iscsi,

and enter the following commands:

# chmod 755 /etc/rc.d/init.d/iscsi

# cd /etc/rc.d/rc2.d

# ln -s /etc/rc.d/init.d/iscsi S50iscsi

# ln -s /etc/rc.d/init.d/iscsi K50iscsi

Then place the sample configuration file in /etc/iscsi/volumegroups, and enter the

appropriate information about your iSCSI-based volume groups.

/etc/rc.d/init.d/iscsi

#!/bin/sh

#

# Sample iSCSI Startup Script for AIX 5.3

#

# This file normally resides as /etc/rc.d/init.d/iscsi, and is

# symbolically linked to /etc/rc.d/rc2.d/S50iscsi and K50iscsi.

#

# Copyright 2007 EqualLogic, Inc.

#

# Version history:

# 1.0 - initial pass GWA

ISCSI_VGFILE=/etc/iscsi/volumegroups

case "$1" in

start)

awk '!/^\#/ { if (NF >= 2) { print "Executing \"varyonvg " $1

"\"..." ; system("varyonvg " $1) ; for (i=2; i<=NF; i+=1) {print

"Executing \"mount " $i "\"..." ; system("mount " $i)}}}' $ISCSI_VGFILE

;;

stop)

awk '!/^\#/ { if (NF >= 2) { for (i=2; i<=NF; i+=1) {print

"Executing \"umount " $i "\"..." ; system("umount " $i)} ; print

"Executing \"varyoffvg " $1 "\"..." ; system("varyoffvg " $1)}}'

$ISCSI_VGFILE

;;

esac

/etc/iscsi/volumegroups

# /etc/iscsi/volumegroups

#

# This file defines the volume groups and associated filesystems so

# that they can be activated and mounted properly during startup,

# and also dismounted and deactivated properly during shutdown.

# The AIX native iSCSI initiator does not handle these functions

# automatically. This file is read by /etc/rc.d/init.d/iscsi,

# which is normally linked to /etc/rc.d/rc2.d/S50iscsi and K50iscsi.

#

# Copyright 2007 EqualLogic Inc.

#

# Format: one entry per line. First field is the name of the

# volume group, additional fields are the mount points. Whitespace

# must separate the fields. Blank lines are ignored.

#

# volume-group mountpoint [mountpoints...]

# iscsi-vg2 /test /test2 /test3

Using the Cisco iSCSI initiator for IBM AIX AIX 5.1 and 5.2 up to ML2

Once the initiator and the required fix have been installed, you are ready to begin using iSCSI

with your AIX system. Please follow the directions in the README file that shipped in the

initiator TAR file, as it contains specific instructions, including required modifications ti

/etc/rc.shutdown to properly unmount iSCSI volumes during shutdown.

The three files that are used to configure iSCSI are:

/etc/iscsi.conf discovery portals and configuration parameters are defined

/etc/iscsi_bindings

device numbers are recorded for persistence and consistency

/etc/filesystems.iscsi iSCSI-based filesystems are automatically mounted and unmounted

Note that the queue depth for the iSCSI hdisk devices will default to 1. The administrator can

change this to improve performance – see Queue Depth above. Also note the Setting Disk

Timeouts section above, although no timeout configuration changes are normally needed.

/etc/iscsi.conf

To work properly with an EqualLogic PS Array, the only information you need to put in

iscsi.conf is the discovery address. This is where the initiator will go to get a list of available

iSCSI targets, and will automatically login to them. Add a line to the file like this:

DiscoveryAddress=172.19.0.1

The address supplied should be the group address, not any of the individual eth port addresses on

your array(s). If you have multiple groups, you can specify multiple discovery addresses here,

and the system will attach to available iSCSI targets on all of them.

There are many other commands and directives available in the iscsi.conf file. It is self-

documenting, so if you have a need for other options (CHAP authentication, for instance), please

see the file itself for instructions on further configuration. For simple operation, setting the

discovery address is sufficient. There are a number of other options available, but we do not

recommend configuring them unless you know you have a specific need. We do not require any

of them.

iscsi-reload

Once you have defined your target portal, you need to instruct the initiator to reread its

configuration files and take appropriate actions to use any new resources that become available.

To do this, you execute the command:

# iscsi-reload

This instructs the initiator to read in new information, while preserving any existing connections.

Other commands available are “/etc/iscsi.clean rem” (which unloads the initiator

completely, dropping all connections), and “/etc/rc.iscsi” (which starts the initiator from

scratch). Note that stopping the initiator disconnects from all targets immediately. This should

not be done lightly, as it can cause real problems for your host if there was a process accessing

any data on one of those volumes. However, it may occasionally be necessary to force the

initiator to completely flush its configuration and start over. Generally speaking, a host reboot

may be more appropriate, and certainly safer.

iscsi-ls

Once you have reloaded the initiator, your configured volume(s) should be visible on your

system. The command “iscsi-ls” is used to interrogate the iSCSI system.

# iscsi-ls -l

*****************************************************************************

Cisco iSCSI Driver Version ... 3.3.2 (25-Jul-2003)

*****************************************************************************

NAME : iqn.1987-05.com.equallogic:00...6-volname

TARGET ADDRESS : 172.19.0.100:3260,1

TARGET ID : 0

STATUS : Connected 172.19.0.25:29174<->172.19.0.100:3260

at Wed Dec 28 08:01:24 2005

SESSION : ISID 00023d000001 TSID 02

LUN 0 : hdisk2

DEVICE STATUS : AVAILABLE

DEVICE DETAILS :

VENDOR : EQUALLOGIC

PRODUCT : PS-200E

PRODUCT REVISION: 0073

DISK CAPACITY : BLOCKCOUNT 0x21eb38f, BLOCKLEN 0x200,

DISKSIZE 17366.45 MB

The most important thing that this report tells us the the device name for the iSCSI volume.

Look at the “LUN” line, and note the name there (hdisk2). PS Arrays provide only a single lun

per target, and so each iSCSI volume will appear as a single hdisk device. (For EqualLogic PS

Arrays, you can identify the volume being used by looking at the end of the “NAME” field - in

this case, “volname.”)

/etc/iscsi_bindings

Once you have successfully attached to a volume, the target number is recorded in the file

“/etc/iscsi_bindings.” This ensures that, on subsequent boots when the order of discovery

might have changed, a particular target is assigned the same device node. Iscsi_bindings is a

plain text file and can be edited by hand. Its content will look something like this:

# iSCSI bindings, file format version 1.0.

# NOTE: this file is automatically maintained by the iSCSI daemon.

# You should not need to edit this file under most circumstances.

# If iSCSI targets in this file have been permanently deleted, you

# may wish to delete the bindings for the deleted targets.

#

# Format:

# bus target iSCSI

# id id TargetName

#

0 0 iqn.2001-05.com.equallogic:6-8a0900-ff4000301-7e1ff088ce9430dd-volname

The file will contain one line for each target that it has connected to. The most useful reason for

editing it is to remove old connections that are no longer used. There is a limit of 15 targets, so

if you have attached to and then discarded 15 volumes, your 16th

will fail because the table is full.

You will want to edit this file and remove the entries for the volumes you will no longer be

accessing.

Once you have attached to your iSCSI volume, you can treat it as an ordinary hard drive, with

one exception (see /etc/filesystems.iscsi below).

Once the iSCSI volume has been bound to an hdisk device, you can manipulate it as you would

any other physical disk.

/etc/filesystems.iscsi

To mount your new filesystem at boot time, you cannot simply add it to the system’s

/etc/filesystems. filesystems is scanned before the network system is operational, so it

will always fail to find and mount your iSCSI-based filesystems. To get around this, the iSCSI

system uses its own table, called /etc/filesystems.iscsi. To properly activate an iSCSI-

based volume group and mount its filesystems at boot time, you must use both

/etc/filesystems, to define the mount options, and /etc/filesystems.iscsi to tell the

iSCSI system to activate and mount those filesystems after the iSCSI system has started.

The entry for /etc/filesystems should describe the filesystem normally, as you would for any

direct-attached storage, except that mount should be false, so that AIX doesn’t try to mount it

when it’s mounting direct-attached storage.

/mount-point:

dev = /dev/lv00

vfs = jfs

log = /dev/loglv01

mount = false

options = rw

account = false

Then you need to add a single line to /etc/filesystems.iscsi in order to cause the system to

mount any logical volumes after activating the new volume groups. The entry will simply be the

LV device, which the iSCSI system will use to enable the enclosing volume group, and then

mount the filesystem using the information in /etc/filesystems. The entries will look like:

/dev/lv00

/dev/lveql01

And now your system is configured properly and ready to go. There are many parameters and

operations in the AIX LVM system that are well beyond the scope of this document. This is

intended just to get you started.

iscsiactlun

The iscsiactlun command dynamically rescans the device information for any session that has

already been established. This will not cause the system to discover any new targets, but it will

permit the system to determine any changes to a volume (most importantly, a change in size) that

is already being used.

Mounting a Snapshot

To mount a Snapshot, you can either mount the snapshot in place of the original volume, or

mount both the base volume and the snapshot at the same time.

Mount the snapshot in place of the original volume

Unmount all file system(s) on iSCSI volumes that are on the primary AIX volume group, and

deactivate the primary volume group using “varyoffvg”. Then perform the following:

o From the AIX server, stop the initiator first by running the “/etc/iscsi.clean rem”

command.

o From the PS Series Storage Array, set the volume offline

o From PS Series Storage Array set the snapshot online

o From the AIX server start the initiator by running the # /etc/rc.iscsi command

o Activate the primary volume group.

o Mount any required file systems.

Mount both the base volume and the snapshot at the same time

o On the PS Series group, set the snapshot online

o Perform the following commands from the AIX server to load the new online snapshot: # /sbin/init.d/iscsi-reload

# iscsiactlun

# lspv

# iscsi-ls -l

o Mount any required file systems.

Extending a Volume Group

To extend an AIX volume group:

o Create a new volume in the group.

o Execute the following comands so that the new volume is visible to the system: # iscsi-reload

# iscsiactlun

# lspv

# iscsi-ls -l

o Extend the volume group using the appropriate smit commands.

Stopping and Starting the Initiator Service

If required, use these commands to stop and restart the iSCSI system on the AIX system when

taking a volume offline:

o From the AIX server stop the initiator first by running the “/etc/iscsi.clean rem”

command

o On the PS Series Storage Array set the volume offline

o From the AIX server start the initiator by running the “/etc/rc.iscsi” command

Using the IBM 1 Gigabit iSCSI TOE Adapter

Although EqualLogic has not tested and qualified this initiator, we can provide some guidance

on getting it properly installed on your AIX systems. However, we can make no guarantees

about its specific compatibility with EqualLogic storage arrays.

Please verify that the BIOS and firmware of the TOE card are fully up to date before proceeding.

IBM's documentation for setting up the iSCSI TOE card can be found here.

Configuration of the IBM iSCSI TOE card is performed in SMIT. Each port is a separate SCSI

interface, and must have its own separate configuration. The first port is given the name "ics0",

the second "ics1", and so on. Each port must havre a separate IP address, and will consult a

separate target configuration file. They can, however, use the same CHAP secrets file if

appropriate.

To configure an iSCSI port, run "smit iscsi", choose "iSCSI Adapter", and then "Change /

Show Characteristics of an iSCSI Adapter".

iSCSI Adapter

Move cursor to desired item and press Enter.

List All iSCSI Adapters

Change / Show Characteristics of an iSCSI Adapter

Generate Error Report

Trace iSCSI Adapter

+--------------------------------------------------------------------------+

| iSCSI Adapter |

| |

| Move cursor to desired item and press Enter. |

| |

| ics0 Available 00-09 iSCSI Adapter |

| ics1 Available 09-09 iSCSI Adapter |

| |

| F1=Help F2=Refresh F3=Cancel |

| F8=Image F10=Exit Enter=Do |

F1| /=Find n=Find Next |

F9+--------------------------------------------------------------------------+

Select the particular port you wish to configure, and hit return. You will see the following

configuration page:

Change / Show Characteristics of an iSCSI Adapter

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

iSCSI Adapter ics0

Description iSCSI Adapter

Status Available

Location 00-09

iSCSI Initiator Name []

Maximum number of commands to queue to the adapter [200] +#

Maximum transfer size [0x100000] +

Discovery Filename [/etc/iscsi/targetshw]

Discovery Policy SLP +

Automatic discovery secrets filename []

Adapter IP Address []

Adapter Subnet Mask []

Adapter Gateway Address []

Apply change to DATABASE only no +

F1=Help F2=Refresh F3=Cancel F4=List

F5=Reset F6=Command F7=Edit F8=Image

F9=Shell F10=Exit Enter=Do

Please assign the adapter an iSCSI Initiator Name first. The only requirement for the iSCSI

Initiator Name is that it begin with "iqn.", and be globally unique; however, it's common to

include the hostname and other descriptive information in the name. For instance, "iqn.aix-

server1-port0".

Since we recommend a queue depth of 128 for every iSCSI hdisk device, we recommend setting

the "Maximum number of commands to queue to the adapter" parameter to 128 times the number

of volumes being connected. For instance, if you are connecting to three volumes, set this

parameter to 384.

Although the Discovery Filename defaults to "/etc/iscsi/targetshw" (where "hw" means this

is the targets file for a hardware initiator), because of the possibility of multiple TOE card ports,

we recommending setting it to "/etc/iscsi/targetshw0" for ics0, hw1 for ics1, and so on. If

you are using CHAP, please consult the IBM documentation for configuration, and set the

"Automatic discovery secrets filename" parameter appropriately.

Set the Discovery Policy to "FILE", and assign this HBA an IP address, subnet, and, if

appropriate, a default gateway. Remember that this TOE card is not an ethernet adapter, and will

not provide you general TCP/IP services. So you cannot, for instance, ping the array over your

TOE card, or telnet to it. If you desire, you can certainly configure one of your normal ethernet

interfaces to also be on the same subnet as the iSCSI traffic if you wish to manage your array

from your AIX server.

Your configuration should now look something like this:

Change / Show Characteristics of an iSCSI Adapter

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

iSCSI Adapter ics0

Description iSCSI Adapter

Status Available

Location 00-09

iSCSI Initiator Name [iqn.aix-server1-port0]

Maximum number of commands to queue to the adapter [256] +#

Maximum transfer size [0x100000] +

Discovery Filename [/etc/iscsi/targetshw0]

Discovery Policy file +

Automatic discovery secrets filename []

Adapter IP Address [172.25.6.174]

Adapter Subnet Mask [255.255.255.0]

Adapter Gateway Address []

Apply change to DATABASE only no +

F1=Help F2=Refresh F3=Cancel F4=List

F5=Reset F6=Command F7=Edit F8=Image

F9=Shell F10=Exit Enter=Do

Hit Enter to commit the changes. If the command fails with the following error, then you must

go back to the iSCSI Adapter page and set the "Apply change to DATABASE only" parameter to

"Yes." That will cause the changes to be applied only at reboot. Then you can reboot to get your

iSCSI Adapter operational.

Method error (/usr/lib/methods/chggen):

0514-029 Cannot perform the requested function because a

child device of the specified device is not in a correct state.

It's fine to configure all ports this way before having to restart the AIX server.

Once you have configured your iSCSI Adapters, you must connect them to their respective

targets. The process is identical to connecting the AIX native software initiator to its targets,

with the two exceptions that the device to scan in cfgmgr is "ics0" (or "ics1" and so on),

instead of "iscsi0"; and the configuration file will be /etc/iscsi/targetshw0 (or

/etc/iscsi/targetshw1, and so on).

Please see the /etc/iscsi/targets section for information about writing the targets configuration

file(s); Adding drives using cfgmgr for information about rescanning for targets; Queue Depth

for setting the Queue Depth on an iSCSI-based hdisk device (and make sure you do that before

using the disk in any way); and Setting Disk Timeouts for information about setting appropriate

timeouts on the devices. Note that Deleting iSCSI Devices also applies.

Since the iSCSI Adapter does not depend on any networking services, the devices connected

through your TOE card ports are always available. You do not have to do anything special to get

them activated and mounted during start-up. Treat them like locally direct-attached storage.

Dell Confidential  For Authorized Use Only   

Document Control and History Information 

Revision  Change Author/Organization  Reason Approved 

By/Organization Date 

A02  M. Cynewski / GCSS  ISO Compliance  M. Gauthier / GCSS  9/2/2009 

A01  Unknown  Initial Release  Unknown  4/28/2008