enabling local and remote high performance backups shyamsundar r software engineer, sms novell, inc....

Enabling Local and Remote High Performance Backups

Shyamsundar RSoftware Engineer, SMSNovell, Inc.

Vijai Babu MadhavanSoftware Engineer, SMSNovell, Inc

© March 9, 2004 Novell Inc.2

one Net: Information without boundaries…where the right people are connected with the right information at the right time to make the right decisions.

The one Net vision

Novell exteNd™

Novell Nsure™

Novell Nterprise™

Novell NgageSM

:

:

:

:


The one Net vision

Novell Nterprise is an innovative family of products which gives you the power to enable and manage the constant interaction of people with your business systems — regardless of who they are or where they are.

Novell Nterprise™

Novell exteNd™

Novell Nsure™

Novell Nterprise™

Novell NgageSM

:

:

:

:


Agenda

Technology overview

SMS technology advancements

Troubleshooting backup on NetWare

Making the most out of your storage

Related technology


Backups

Backup is a second class citizen…


Backup data flow

TSAFS

*

Backup Engine

File Systems

Local backup data flow


Backup data flow

TSAFS

Backup Engine

File Systems*



Backup data flow

TSAFS

Backup Engine

File Systems

*



Backup data flow

SMDRBackup Engine SMDR

Network

*

TSAFS

Backup Engine

File Systems

Remote backup data flow

TSAFS

File Systems


Backup data flow


Network

*Remote backup data flow

TSAFS

File Systems

TSAFS

Backup Engine

File Systems


Backup data flow


Network


TSAFS

File Systems

TSAFS

Backup Engine

File Systems


Backup data flow


Network


TSAFS

File Systems

Backup Engine

TSAFS

File Systems

16

Local backup data flow: Systemic view

CPU-memory bus

Cache

CPU

Bridge

Storage (SCSI) Bus

Disk

I/O (PCI) bus

HBA

Memory

*

17


CPU-memory bus

Cache

CPU

Bridge

Storage (SCSI) Bus

I/O (PCI) bus

HBA

Memory

*

Disk

18


CPU-memory bus

Cache

CPU

Bridge

Storage (SCSI) Bus

I/O (PCI) bus

HBA

Memory

*

Disk

19


CPU-memory bus

Cache

CPU

Bridge

Storage (SCSI) Bus

I/O (PCI) bus

HBA

Memory

*

Disk

20


CPU-memory bus

Cache

CPU

Bridge

Storage (SCSI) Bus

I/O (PCI) bus

HBA

Memory*

Disk

21


CPU-memory bus

Cache

CPU

Bridge

Storage (SCSI) Bus

I/O (PCI) bus

HBA

Memory

*

Disk

22


CPU-memory bus

Cache

CPU

Bridge

Storage (SCSI) Bus

I/O (PCI) bus

HBA

Memory

*

Disk

23


CPU-memory bus

Cache

CPU

Bridge

Storage (SCSI) Bus

I/O (PCI) bus

HBA

Memory

*

Disk

24


CPU-memory bus

Cache

CPU

Bridge

Storage (SCSI) Bus

I/O (PCI) bus

HBA

Memory

*Disk

25

Remote backup data flow: Systemic view

CPU-memory bus

Cache

CPU

Bridge

I/O (PCI) bus

HBA

Memory

Network

Network Controller

Storage (SCSI) Bus

Disk*

26


CPU-memory bus

Cache

CPU

Bridge

I/O (PCI) bus

HBA

Memory

*

Network

Network Controller

Storage (SCSI) Bus

Disk

27


CPU-memory bus

Cache

CPU

Bridge

I/O (PCI) bus

HBA

Memory

*

Network

Network Controller

Storage (SCSI) Bus

Disk

28


CPU-memory bus

Cache

CPU

Bridge

I/O (PCI) bus

HBA

Memory*

Network

Network Controller

Storage (SCSI) Bus

Disk

29


CPU-memory bus

Cache

CPU

Bridge

I/O (PCI) bus

HBA

Memory

*

Network

Network Controller

Storage (SCSI) Bus

Disk

30


CPU-memory bus

Cache

CPU

Bridge

I/O (PCI) bus

HBA

Memory

*

Network

Network Controller

Storage (SCSI) Bus

Disk

31


CPU-memory bus

Cache

CPU

Bridge

I/O (PCI) bus

HBA

Memory

*Network

Network Controller

Storage (SCSI) Bus

Disk

32


CPU-memory bus

Cache

CPU

Bridge

I/O (PCI) bus

HBA

Memory

*Network

Network Controller

Storage (SCSI) Bus

33


CPU-memory bus

Cache

CPU

Bridge

I/O (PCI) bus

HBA

Memory

Network

Network Controller

Storage (SCSI) Bus

*

34


CPU-memory bus

Cache

CPU

Bridge

I/O (PCI) bus

HBA

Memory

*

Network

Network Controller

Storage (SCSI) Bus

35


CPU-memory bus

Cache

CPU

Bridge

I/O (PCI) bus

HBA

Memory

*Network

Network Controller

Storage (SCSI) Bus


Available network bandwidth

Ethernet wire capacity in theory• 100 Mbps = 94.44 Mb/s ~= 676 MB/min

– Accommodating space loss due to protocol headers

– Accommodating time loss due to inter-frame gap• 1000 Mbps ~= 6755 MB/min

Practical dedicated Ethernet wire capacity• 100 Mbps – ~656 MB/min – 97%• 1000 Mbps – ~6417 MB/min – 95%

“Ideal bulk transfer applications should be bandwidth bound”


Available network bandwidth

Ethernet wire capacity in theory• 100 Mbps = 94.44 Mb/s ~= 676 MB/min

– Accommodating space loss due to protocol headers

– Accommodating time loss due to interframegap• 1000 Mbps ~= 6755 MB/min

Practical dedicated Ethernet wire capacity• 100 Mbps – ~656 MB/min – 97%• 1000 Mbps – ~6417 MB/min – 95%

“Ideal bulk transfer applications should be bandwidth bound”


Agenda

Technology overview




Related technology


How fast can remote backups go?

We can do 1TB in 3.5 Hours on the wire at 4.8GB/Min, with the streaming SMDR prototype


Streaming SMDR prototype results

“Remote matching local performance...”

Setup#1 Setup#2 Setup#3

470 1080 2050

510 1100 2100 Streaming PrototypeLocal

05/24/1901 01/04/1903 09/30/190504/14/1901 12/15/1902 08/11/1905

0

500

1000

1500

2000

2500

Thr

ough

put

(MB

/Min

)

8.5 1.9 2.4

Streaming PrototypeLocal% Difference

Setup #1: 2GB SYS volume with an average file size of 27KB


Setup #3: 6GB volume with 6000 files each being 1MB in size





470 1080 2050


05/24/1901 01/04/1903 09/30/190504/14/1901 12/15/1902 08/11/1905

0

500

1000

1500

2000

2500

Thr

ough

put

(MB

/Min

)

8.5 1.9 2.4







Improvements over existing SMDR

0

500

1000

1500

2000

2500M

B /

min Existing SMDR

Prototype 1

Prototype 2

Existing SMDR 212 305 560

Prototype 1 216 440 1550

Prototype 2 470 1080 2050

Setup 1 Setup 2 Setup 3



“Improved remote restore performance”

0

500

1000

1500

2000

2500

3000M

B /

min Existing SMDR

Prototype 1

Prototype 2

Existing SMDR 600

Prototype 1 1800

Prototype 2 2600


How Streaming SMDR works

Alleviates latency effects on throughput by anticipation

Keeps the network pipe full

Better network bandwidth utilization

Read ahead, lazy write

Decouples the interface from the implementation


Who gets the benefit

Applications that use SMDR for remote backups

Data movement applications• Novell migration wizard• Novell server consolidation utility• Novell DFS volume move/volume split• Commercial copy applications


SMDR: Features

Existing Features• Provides location transparency• Abstracts communication details, supports both

TCP/IP and IPX/SPX• Advertisement/Name resolution• Synchronous RPC model

Recent Features• Multiple IP address support• DNS support for name resolution• Policy based discovery/name resolution mechanism

order• iManager and server console based configuration


TSAFS: Features

Scalability• Serial nature of backup does not limit scalability• Re-architected TSA to scale with disk configurations

Manageability• Improved manageability, helps identify performance sweet

spots• iManager and server console based configuration

Availability• Cluster enabled, supports continuation of backup after a

fail-over/fail-back• Not just with NCS, even with server restarts

Ships with NetWare® 6.5, NetWare® 6.0 SP4


TSAFS: Features








Sample TSAFS test results

0

2000

4000

6000

8000Th

roug

hput

(MB

/min

)

TSA600 1380 1400

TSAFS 3300 6120

Tweaked 5880 6120

Setup #1 Setup #2

Setup #1: Ultra 160 controller with 4 disks using RAID 5, with PCI-X 64bit 100 MHzSetup #2: Ultra 160 controller with 6 disks using RAID 0, with PCI-X 64bit 100 MHz

TSAFS improvements and tweaking effects


Sample TSAFS test results

TSAFS improvements and tweaking effects

Setup #1: Ultra 160 controller with 4 disks using RAID 5, with PCI-X 64bit 100 MHzSetup #2: Ultra 160 controller with 6 disks using RAID 0, with PCI-X 64bit 100 MHz

0

2000

4000

6000

8000Th

roug

hput

(MB

/min

)

TSA600 1380 1400

TSAFS 3300 6120

Tweaked 5880 6120

Setup 1 Setup 2


Agenda

Technology overview




Related technology


General Guidelines

Rules of Thumb• Eliminate one component at a time• Try and eliminate broader sub systems first

– For e.g., target the disk or network first

Use tools• DPMETERM• TSATEST• NetWare Remote Manager (NRM)• iManager and Server Console


Troubleshooting demo

Establish a baseline• TSATEST• Backup application

Can it be improved?• DPMETERM

Can we go beyond this?• Using NRM• Using TSATEST• Tweaking TSAFS, ReadThreadsPerJob

and ReadBufferSize






and ReadBufferSize


Troubleshooting SMDR issues

Use tools for performance issues• PERFTEST• NETMON• Other packet capture/analysis utilities

Discovery/Name Resolution issues• SLP• SMDR NEW• SMDR CONN

“Ensure simultaneous remote backups do not flood your network”


Troubleshooting TSAFS issues

TSAFS debug log• TSAFS /SmsDebug=<Value> /SmsDebug2=<Value>

Server memory usage• TSAFS /CacheMemoryThreshold=<Percentage>

Other factors• Competing software (Compression/Anti-Virus)• CPU usage/Busy threads, NRM I/O statistics


Agenda

Technology overview




Related technology

67

Power of aggregation

CPU-memory bus

D

PCI bus

D D D D

D D D D D

D D D D D

D D D D D

D D D D D

D D D D D

D D D D D

D D D D D

D D D D D

D D D D D

D D D D D

D D D D D

SCSI/RAID Controller

SCSI Controller

PCI bus...

...

PCI busPCI busPCI bus ......

Fibre Channel HBA


Suggestions on exploiting bandwidth

Create parallel I/O paths for high performance• Load balance peer-to-peer PCI buses

– Distribute your storage and network controllers

• Use multiple channels per HBA and stripe across them

Use technology to suit performance needs• 1Gbps networks are becoming common, 4Gbps cards are out

in the market, 10Gbps is close to reality

Mix and match with care• Do not use Ultra 320 disks with an Ultra 160 SCSI controller• Performance is limited by the lowest bandwidth component in

your data pipe

Design your storage configuration for high availability as

well









your data pipe


well


Agenda

Technology overview




Related technology


Related Technology

File Copy On Write• Permits open file backup

Pool Snapshots• Point in time data snapshots

Freeze/Thaw• Permits data consistency across compliant

applications

Versioning• User managed versioning of files, easing restores


Related Technology




applications



SMS on Linux

Location Transparency

Enterprise scale, logical backup

Multiple file system support

Rich meta-data support


SMS: In Summary

One-stop Network wide Storage Management Solution

• High Performance, Scalable

• Extensible solution

• Multi-Latency/Bandwidth

environments

• True backup virtualization


Backups

Backup is a second class citizen…

Or is it?

General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. Novell, Inc., makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.

No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

enabling local and remote high performance backups shyamsundar r software engineer, sms novell, inc....

Documents