desmystifying extended distance ficon (aka persistent iu pacing), por dr. steve guendert

Demystifying Extended Distance FICON (aka Persistent IU Pacing)

Dr. Steve Guendert

Principle Engineer/Global Solutions Architect

Brocade Communications

04/11/2023© 2011 Brocade Communications Systems, Inc. Company Proprietary Information 2

Legal Disclaimer

All or some of the products detailed in this presentation may still be under development and certain specifications, including but not limited to, release dates, prices, and product features, may change. The products may not function as intended and a production version of the products may never be released. Even if a production version is released, it may be materially different from the pre-release version discussed in this presentation. NOTHING IN THIS PRESENTATION SHALL BE DEEMED TO CREATE A WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, INCLUDING BUT NOT LIMITED TO, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT OF THIRD-PARTY RIGHTS WITH RESPECT TO ANY PRODUCTS AND SERVICES REFERENCED HEREIN. Brocade, the B-wing symbol, BigIron, DCFM, DCX, Fabric OS, FastIron, IronView, NetIron, SAN Health, ServerIron, TurboIron, and Wingspan are registered trademarks, and Brocade Assurance, Brocade NET Health, Brocade One, Extraordinary Networks, MyBrocade, VCS, and VDX are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned are or may be trademarks or service marks of their respective owners.


Abstract

• In 2008 IBM announced IU pacing enhancements that allow customers to deploy z/OS Global Mirror (zGM) over long distances without a significant impact to performance. This is more commonly known by the marketing term Extended Distance FICON. The more technically accurate term as defined in the FC-SB3/4 standards is persistent IU pacing. How this functionality works, and what it actually does for the end user has remained a mystery, thanks in large part to the marketing term used. This paper/presentation will demystify how the technology works, and how it can benefit the end user.

CMG 2011 paper submission


Agenda

• Basics: What is Extended Distance FICON (and what it is not)?

• Sequences, Open Exchanges, and Information Units (oh my!)• IU Pacing and Persistent IU Pacing• How does it really function?• Performance Implications

– Do I still need channel extension?

• Concluding thoughts and observations


Basics of Extended Distance FICON

© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

Extended Distance FICON

• Extended Distance FICON is an enhancement to the industry-standard FICON architecture (FC-SB-3) that can help avoid degradation of performance at extended distances by implementing a new protocol for "persistent" Information Unit (IU) pacing

• Some storage vendors will position the Extended Distance FICON feature as eliminating the need for “Those expensive channel extenders”

The rumor of the demise of channel extension is greatly exaggerated

04/11/2023 6


What is Extended Distance FICON?

• Announced by IBM 26 February 2008• It is more of a marketing term than a technological term.• IBM announced IU pacing enhancements for z/OS Global

Mirror (zGM)• The announcement really announced that the z10 and

DS8000 supported the new FC-SB3 amendments for Persistent IU Pacing.


What Extended Distance FICON is (and is not)

• It is a performance enhancement mechanism for FICON I/O traffic traversing “long” distances.

• It really does not increase the distance the FICON I/O traffic can go.

• Distance capabilities depend on other things– Buffer to buffer credits


Hardware requirements

• System z196• System z10

– Driver 73 with MCL F85898.003– Driver 76 with MCL N10948.001

• DASD– IBM DS 8000/8700/8800– EMC VMAX with Enginuity 5874 Q4 service release – HDS USP-V


Sequences, Open Exchanges, and Information Units

(Oh my!)


The FICON Upper Layer Protocol (ULP)

• Provides 3 resources for managing the underlying physical credits used to transmit the actual data in an I/O.

– FC-SB3/4 Sequences– Exchanges (OEs)– Information Units (IUs)

• A credit is a 2K physical data transmission buffer– More commonly known as buffer to buffer credits (BB credits)– Used for flow control between two optically adjacent fibre channel ports.


Sequences

• The FC-PH standard provides a mechanism that assembles sub-blocks of data contained in the payloads of one or more frames into a single data block when receiving frames.

• Each FC-4 ULP defines what the contents of the sequences are for its specific ULP functions.

– FICON: IUs


Information Units (IUs)

• An IU is a logical resource comprised of 1-4 credits• Also synonymous with a Channel Command Word (CCW)• FICON channels support a maximum of 16 concurrent IUs for

each Open Exchange• IUs contain commands, device-level controls, link controls,

and other related functions• Common examples of IUs:

– Command IUs– Solicited/unsolicited data IUs– Solicited/unsolicited control IUs

Command Mode


Open Exchanges (OEs)

• An Open Exchange (OE) is a logical resource representing an I/O being processed by a channel.

• FICON and FICON Express/Express 2 channels on zSeries servers can support 32 concurrent OEs.

• FICON Express 2/4/8 channels on System z9 and later servers can support 64 concurrent OEs.

• IUs sent by a FICON channel during the execution of an SB-3 link-control function or device-level function are restricted to one exchange, known as an outbound exchange.

• IUs which a FICON channel receives during an I/O operation are restricted to a different exchange, known as an inbound exchange.


Inbound and Outbound Exchanges

• An inbound exchange and an outbound exchange transfer IUs in a single direction.

• An existing pair occurs when both an inbound and an outbound exchange simultaneously exist between a channel and control unit for the execution of the same device-level or link-level function.

– The control unit is then said to be connected to the channel.• A channel program, which is executed in a single connection, uses only

one exchange pair. • If the connection is removed by closing the exchanges during the channel

program, a new exchange pair is required to complete the channel program.

Command Mode


IU Pacing and Persistent IU Pacing

How does it really work?

IU pacing

• Each FICON channel port is capable of multiple concurrent data exchanges (currently a maximum of 64) in full duplex mode. Information for active exchanges is transferred in Fibre Channel sequences mapped as FICON Information Units (IUs) which consist of one to four Fibre Channel frames

• As with prior Z channel protocols, there is a concept of a channel to control unit "connection." In its most primitive form, a connection is associated with a single channel program. In practice, a single channel program may result in the establishment of several sequential connections.

• In order to prevent a single channel/control-unit data flow from monopolizing the majority of the FICON channel bandwidth, the FICON protocol uses a pacing algorithm which is negotiated during the ELP-LPE sequence. For most cases, the pacing window is 16 IU’s.

04/11/2023© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

17


IU Pacing

• FC-SB-3 Rev 1.6 defines an IU pacing protocol that controls the number of IUs that can be in flight from a channel to a control unit.

• IU pacing is an FC-SB-3 level-4 function that limits the number of Channel Command Words (CCWs), and therefore the number of IUs, that can either transmit (write) or solicit (read) without the need for additional control-unit-generated acknowledgements called command responses.

• The control unit may increase the pacing count (the number of IUs allowed to be in flight from channel to control unit) in the first Command Response IU sent to the channel.

• The increased pacing count is valid only for the remainder of the current outbound exchange. In certain applications, at higher link speeds and at long distances, a performance benefit is obtained by the increase in the allowed pacing count.

Basics


IU Pacing

• IU pacing is not a flow control mechanism.– Flow control is adequately addressed by the FC-PH level buffer-to-

buffer crediting function.

• IU pacing is a mechanism intended to prevent I/O operations that might introduce very large data transfers from monopolizing access to Fibre Channel facilities by other concurrent I/O operations.

– Essentially it provides a load-sharing or fair-access mechanism for multiple competing channel programs.

What it really is


IU Pacing

• The first burst of IUs from the channel to the control unit can be no larger than a default value of 16.

• This causes a delay in the execution of channel programs with more than 16 commands at long distances because a round trip to the control unit is required before the remainder of the IUs can be sent by the channel, upon the receipt of the first command response, as allowed by the increased pacing count.

• Pacing credits can be adjusted dynamically from these values by control unit requests for specific protocol sequences; however, the channel is not bound to honor control unit requests for larger IU pacing windows.

Limitations


IU Pacing

• It yields mixed results– insures more predicable I/O response times on heavily loaded channels.– produces less optimal results for very-long-distance deployments.

• Increased link latencies can introduce dormant periods on the channel and its Wide Area Network (WAN) link.

– Dormant periods occur when delays waiting for anticipated command responses increase to the point where the pacing window prohibits the timely execution of CCWs that might otherwise be executed to insure optimal performance.

• Mixed results, and limitations led to new enhancement

Mixed results?


Persistent IU Pacing

• Persistent IU pacing is a method for allowing FICON channels to have more IUs in flight, and retain a pacing count that may be used at the start of execution of a channel program.

• This may improve the performance of long I/O programs at higher link speeds and long distances by allowing the channel to send more IUs to the control unit, thereby eliminating the delay of waiting for the first command response.

• The channel retains the pacing count value, presented by the control unit in accordance with the standard, and uses that pacing count value as its new default pacing count for any new channel programs issued on the same logical path.

What is it?


Persistent IU Pacing

• The control unit must support the new IU pacing protocol, which is detected during the Establish Logical Path/Logical Path Established (ELP/LPE) sequence.

– The channel will default to the current pacing values when operating with control units that cannot exploit persistent IU pacing.

• Control units that exploit the enhancement to the architecture can increase the pacing count (the number of IUs allowed to be in flight from channel to control unit).

– This can allow the channel to remember the last pacing update for use on subsequent operations to help avoid degradation of performance at the start of each new operation

Exploiting the technology


Persistent IU Pacing Functionality

• Each FICON channel provides an IU pacing credit that is initialized at either the start of each channel program or at the reconnection to continue the execution of a channel program.

– Remember that the IU pacing credit is the maximum number of IUs that a channel may send on a given outbound exchange before it receives a command-response IU.

– A FICON channel may operate in the default IU pacing mode or in the persistent IU pacing mode.

– A FICON channel operates in persistent IU pacing mode on each logical path for which both the channel and control unit indicate support for the persistent IU pacing optional feature.

– On any logical path for which the persistent IU pacing optional feature is not supported, a FICON channel operates in the default IU pacing mode.



• When a FICON channel is operating in default IU pacing mode, the IU pacing credit in effect at the start of a channel program shall be set to a model-dependent (DASD array specific) value no greater than the default value of 16.

• When a FICON channel is operating in persistent IU pacing mode, the IU pacing credit is initialized with the current pacing credit in effect for the logical path.

At the channel



• When a control unit is operating in persistent IU pacing mode, the control unit may request that the IU pacing credit for the logical path be modified at the start of a channel program.

– Any modification of the IU pacing credit shall take effect at the start of the next channel program on the logical path and shall persist until changed again by the control unit, or until a system reset is performed for the logical path.

– The control unit may request that the IU pacing credit be modified by providing an IU pacing parameter in either the command response or in the status sent in response to the first command of a channel program.

• A control unit may increase, decrease, or reset the pacing credit to the default value. A pacing parameter value of zero shall have the effect of resetting the credit to the default value of 16.

At the control unit



• In order to avoid resetting the pacing count to the default value, the control unit must retain its desired setting and include this value in the pacing parameter for all Command Response IUs and Status IUs in which the pacing parameter is valid.

– This must be done for all device operations on the same logical path.

• If the control unit sets the IU pacing parameter to a value less than or equal to the default value, the FICON channel shall not increase the IU pacing credit above the default value.

• If the control unit sets the IU pacing parameter to a value greater than the default value, then the FICON channel may increase the IU pacing credit by any amount up to the value indicated by the IU pacing parameter.

At the control unit


Performance ImplicationsDo I still need channel extension technology?

04/11/2023 29

Why extension of storage networks?

• Safeguard critical business data• Provide near continuous access to applications and services

in the event of a localized disaster


04/11/2023 30

High Level Design Considerations

• Classification of your stored data– How important are the subsets of data for your business operations?

• How often do you need to backup this data?• In event of failure, how quickly do you need to recover?• Recovery Point/Time Objectives (RPO/RTO)

– Synchronous/Near-Synchronous?– Asynchronous?– Electronic tape vaulting?

For your extended storage network


04/11/2023 31

Objectives of extending FICON channels

• Protect the integrity of the data transaction• Keep the FICON channel busy doing useful work• Reduce the number of round trips for channel transactions



Z/OS Global Mirror (formerly XRC)

• Configurations typically utilize channel extension technology between the remote host and the primary controller.

– The channel extension technology in these configurations typically uses advanced emulation/acceleration techniques to analyze the CCW chains, and modify them to perform local acknowledgements so as to avoid throughput performance degradation (commonly known as “data droop”) caused by extra round trips in the long distance environment.• This technology is typically an additional software license added to the channel

extension hardware.

Channel extension emulation

04/11/2023 33

FICON XRC Emulation over Distance

• IBM z/OS Global Mirror (formerly XRC) data replication is performed by pulling information from a primary data store by SDM (remote read operation)

• Distance (latency) impacts read performance• When FICON emulation is enabled,

Brocade 7800 and FX8-24 automatically go into emulation mode

• Emulates XRC RRS (Read Record Set) sequences, accelerating data flows to the host, and queuing at the channel for SDM

• Maintains integrity of command and acknowledgement sequences

Brocade FX8-24 (or 7800)

Z-Series MF Hosts

PrimaryDASD

Target/SecondaryDASD

IP Network

Brocade 7800 (or FX8-24)

Primary

Datacenter

Recovery

Datacenter

XRC System Data Mover



Z/OS Global Mirror (formerly XRC)

• Persistent IU pacing produces performance results similar to XRC Emulation at long distances.

• It does not really extend the distance supported by FICON, but can provide the same benefits as XRC Emulation.

• In other words, there may be no need to have XRC Emulation running on the channel extension hardware.

Do we need emulation technology?


Some performance comparisons


XRC Extended Distance FICON Channel Extension4K writes to 1 Volume, 4 Enhanced SDM Readers

36

3200km, Brocade emulation OFF+Pacing

3200km, Brocade emulation OFF

3200km, Brocade emulation ON







Direct FICON

0 5 10 15 20 25 30 35

I/O Ranges Where EmulationProvides Greater Capacity Than Persistent IU Pacing

Application I/O Rate (thousand I/Os per second)


Extended Distance FICON Single-ReaderXRC Extended distance FICON channel extension 4k writes to 1 volume, single reader

37










Direct FICON

0 2 4 6 8 10 12 14 16 18

I/O Ranges Where EmulationProvides Greater Capacity Than Persistent IU Pacing

Application I/O Rate (thousand I/Os per second)



• The workload consists of 64 jobs performing 4 KB sequential writes to 64 data sets with 1113 cylinders each, which all reside on one large disk volume.

• There is one SDM configured with a single, non-enhanced reader to handle the updates.

• When turning the XRC Emulation off the performance drops significantly, especially at longer distances.

• However, after the persistent IU pacing functionality is installed on the IBM DASD array, the performance returned to where it was with XRC Emulation on, except at very high I/O rates.

• For this case, Persistent IU pacing allows the same functionality and performance benefits without the need for the additional license.

• For very high I/O rates, the emulation technology outperforms the perisistent IU pacing results.

persistent IU pacing performance comparisons for a sequential write workload



• Achieved for the z/OS Global Mirror (XRC) SDM “Read Record Set” (RRS) data transfer from a DASD array (that supports persistent IU pacing) to the SDM host address space.

• The control units that support persistent IU pacing are able to increase the pacing count.

– The pacing count is the number of IUs that are “in flight” from a channel to a control unit. FICON supports IU pacing of 16 IUs in flight.

– Persistent IU pacing increases this limitation for the RRS CCW Chain to permit 240 IUs in flight without waiting for an acknowledgement from the control unit, eliminating handshakes between channel and control unit.

– Improved IU pacing with 240 IU instead of 16 will improve the utilization of the link.

persistent IU pacing performance comparisons for a sequential write workload


IU Pacing and Extended Distance

• Extended Distance FICON removes IU pacing completely and provides for an optimized path to send no more than approximately 240 IUs from the channel to a control unit without waiting for a command response to open the IU pacing window.

• Extended IU Pacing is only operational on FICON XRC connections between a z/OS system and a Extended Distance FICON enabled XRC Primary control unit

– DASD that supports this larger IU pacing value include• IBM DS8xxx• Enginuity 5874 Q4 service release (i.e. requires the VMAX DASD array)• HDS with their USP-V

A good enhancement

40


IU Pacing

• Some of the collateral, including sales presentations by Storage Vendors state that the Extended Distance FICON feature eliminates the need for those expensive channel extenders.

– This could lead a customer to believe the feature address devices and applications other than Global Mirror (which it does not)

– It is also is sometime true; BUT……– Brocade does more for Global Mirror than just address the impact

latency has on the IU pacing protocol• The Brocade solution also emulates DVC-Ack Processing and (under debug flag

control) status accept processing

The real story

41


OK, what do you mean by “Maybe no need”

• Extended Distance FICON is for DASD• What if you are using the extension device(s) for

something else?– remote tape connectivity?

04/11/2023 43

Who Needs FICON emulation technology?

• Mainframe installations who need to extend FICON channels beyond natively supported distances (beyond 100-120km)

– Extend FICON applications over distance via FCIP– Facilitate remote connectivity between

• Primary DASD and remote host running SDM (System Data Mover) for XRC• Host and IBM VTS or TS7700• IBM VTC and VTS for peer-to-peer VTS• Host and remote Oracle VSM controller• Oracle VSM controller and remote tape library/drives• Clustered Oracle VSMs• Host and Teradata systems

• Anyone looking to conserve WAN bandwidth (reducing costs)

Much more use than with just zGM



Concluding Thoughts and Observations

• Persistent IU pacing, AKA “Extended Distance FICON” does not increase the distance support for FICON.

– It enhances the performance of FICON traffic going over long distance• This performance enhancement is intended primarily for use with z/OS Global Mirror

(formerly known as XRC). • This technology does not alter anything with respect to the amount of buffer credits needed

to support long distance FICON data transmission. – Hence, it does not change the need for FICON directors and/or channel extension hardware

used for multi-site disaster recovery/business continuity architectures. • Persistent IU pacing may eliminate the need for XRC emulation technology on the channel

extension hardware and this may reduce the cost of this hardware. • For high I/O rates XRC emulation technology is still a good idea.

– Emulation technology does more for Global Mirror than just address the impact latency has on the IU pacing protocol


Key References

• CMG 2011 Paper “Demystifying Extended Distance FICON”• zJournal August/September 2011 issue.• FC-SB2/3/4 standards• FS-FS 2/3 standards

desmystifying extended distance ficon (aka persistent iu pacing), por dr. steve guendert

Technology