announcing: samqfs** rebrandingto:konferenz-nz.dlr.de/pages/samfs2015/present/1... ·...

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |

Announcing: SAM-‐QFS Rebranding to: Oracle Hierarchical Storage Manager 6.0

Bob Raymond, Director Hardware Engineering Ben Dischinger, Principal SoPware Engineer


Program Agenda

Announcing Oracle HSM 6.0

Oracle HSM 6.0 overview

Patch releases

1

2

3


Announcing Oracle Hierarchical Storage Manager 6.0 • What is changing:

•  SAM-‐QFS 5.4 is now Oracle HSM 6.0 •  FuncSonality of Oracle HSM 6.0 is idenScal to SAM-‐QFS 5.4

– No changes to StorageTek QFS SoPware branding

• Why its changing: – BeXer promote SAM-‐QFS’s posiSon within the Oracle soPware porZolio – Reflect Oracle’s increased investment and commitment to the product – DifferenSate from Oracle’s other archive soPware products – Get in front of new customers


Rebranding -‐ Details •  StorageTek Storage Archive Manager (SAM-‐QFS) now Oracle Hierarchical Storage Manager (Oracle HSM) – Downloads of Oracle HSM 6.0 from e-‐delivery are now available (released 3/3/15) –  Last patch for SAM_QFS 5.4 will be 5.4-‐02 (released 3/3/15). All future patches will be on Oracle HSM 6.0

• Oracle HSM includes the QFS file system

•  The QFS file system can be purchased standalone as well called StorageTek QFS (6.0.0)

• Documents available: hXp://www.oracle.com/technetwork/documentaSon/tape-‐storage-‐curr-‐187744.html#samqfs

hXp://www.oracle.com/technetwork/documentaSon/tape-‐storage-‐curr-‐187744.html#hsmqfs


Release SAM 5.4-‐02 and Oracle HSM 6.0 •  25 bug fixes •  Security fixes •  Two enhancements

– Extended aXribute backup/restore performance improvements – SwiP object archive status API


Oracle HSM 6.0 In Depth


Oracle Hierarchical Storage Manager (Oracle HSM)

•  Mature HSM / archiving product with proven scalability & reliability –  Originally developed in 1995 by LSC Inc., purchased by SUN in 2001 –  Some customers managing 2 billion+ files (in mulSple file systems)

•  Demoed 1 billion files on a single file system using FS1 system1

–  600+ acSve customer installaSons

•  Seamless policy-‐based hierarchical storage management / archiving

•  OpSmized for Solaris, archive and security environments

•  Open standards-‐based –  POSIX compliant file system

–  .tar format for wriSng to archive targets

Complete, Open, Integrated Storage Management

1.  hXp://my.oracle.com/site/pd/sss/soluSons/Sered-‐storage/index.html


Oracle HSM Architecture

Metadata Server

Primary Disk Tape Storage Tier

control path

Data path

Clients

QFS Shared File System

SAM

Capacity Disk Storage Tier

… up to 4 archive targets


QFS File System

• Max ~4 PB Disk Capacity

• Max ~4 billion files

•  100’s of TB file size • Aggregate Bandwidth GBs/sec

• Metadata server (MDS) – Concurrency control – Directory / Inode operaSons

Shared QFS Filesystem

LUN

QFS Client

QFS

LUN LUN LUN …

QFS Client

QFS Client

QFS Client …

MDS

≤252 16 TB LUNs

≤1024 Clients

Metadata


Horizontal Scalability: tape ingest performance improved 10x!

• Use QFS clients for archive and staging to/from tape

OpenStack SwiP support for object based storage

•  Create private clouds leveraging Sered storage

Automated Data Integrity ValidaSon with Self-‐Healing

LTFS import / export support

New Advancements – Oracle HSM 6.0


Distributed Tape I/O


Horizontally Scaling Tape Prior to SAM-‐QFS 5.4

LUN

QFS Client

LUN

LUN

LUN

…

QFS Client

QFS Client

QFS Client

…

Metadata

Tape

Tape

Tape

Tape

… Up to 64 tape drives per library

…

MDS SAM

QFS


Horizontally Scaling Tape Oracle HSM 6.0 Distributed Tape I/O

LUN

QFS Client

QFS

LUN

LUN

LUN

…

QFS Client

QFS Client

QFS Client

…

MDS Metadata SAM

Tape

Tape

Tape

Tape

… Up to 64 tape drives per library

Datamover

Datamover

Datamover

…


Horizontally Scaling Tape

• QFS Clients can now parScipate in tape I/O – Up to 9 Datamover clients (must run Solaris) – SAM on MDS sSll controls:

• All tape library operaSons (loads, ejects) • Building archive requests • Scheduling decisions

– Archive requests, Stage streams (groups of files to copy) • Round robin across datamovers

Oracle HSM 6.0 Distributed Tape I/O



• All tape drives must be visible to datamover clients – Fiber channel switch zoned appropriately

• Configure persistent binding for tape devices – Ensure consistent /dev/rmt device names across hosts – Edit /etc/devlink.tab on each host

Tape Hardware ConfiguraLon



• Edit /etc/opt/SUNWsamfs/defaults.conf – Add ‘disSo = on’

• Edit /etc/opt/SUNWsamfs/archiver.cmd (opSonal) – By default all archive sets will use distributed I/O – Add ‘-‐disSo=off’ to archive set parameters where not wanted

• Run ‘samd config’

Oracle HSM 6.0 MDS ConfiguraLon



• Edit /etc/opt/SUNWsamfs/defaults.conf – Add ‘disSo = on’

• Edit /etc/opt/SUNWsamfs/mcf – Add entry for each tape drive – Use consistent equipment number and ‘-‐’ for family name

• /dev/rmt/60cbn 901 S -‐ on

• Run ‘samd config’

Oracle HSM 6.0 Datamover ConfiguraLon


Cloud Archiving


Building Cloud Infrastructures Support for Cloud

• Use Oracle HSM in an OpenStack SwiP cloud environment

• Store and retrieve data using OpenStack SwiP


Oracle HSM Cloud ConfiguraSons SAM as a Cloud in 5.4 release and in Oracle HSM 6.0

SwiP Server

Cloud users

Metadata Server

Primary Disk Tape Storage Tier

(control path)

Data path

QFS Clients/SwiP storage nodes


Storage Network

Oracle HSM – SWIFT Deployment Diagram (Logical)

REST Swift Proxy

External SWIFT Clients

Up to 512 Nodes

REST

REST

REST

REST

Swift Storage Node QFS Client

SwiP Storage Node QFS Client

Swift Storage Node QFS Client

SwiP Storage Node QFS Client

Oracle HSM MDS

Oracle HSM MDS

External SWIFT Clients External

SWIFT Clients Fire

wal

l


Oracle HSM 6.0 OpenStack SwiP IntegraSon

• “How to Configure OpenStack SwiP with Modern Tape” – hXp://www.oracle.com/goto/hsm

Detailed Whitepaper Available


LTFS Import/Export


How Does LTFS Make a Tape Self-Describing?

METADATA PARTITION

FILE PARTITION

Beginning of Tape End of Tape

A self-describing format is a specification for writing a file and its descriptive metadata to the same storage medium

LTFS FormaXed Tapes Have Two ParSSons: 1-‐ Metadata ParLLon & 2-‐ File ParLLon

1

2

LTFS FORMATTED TAPE


Fibre Flash Disk Tape

Lower Data Transfer & DR Costs With StorageTek LTFS Software

LTFS FORMATTED TAPE

$$$$

Limited

High

$$$

Global

High

$$

Global

Low

$

Global

High

Cost:

Availability:

Reliability:

“Tape is more suitable for physical transportaSon of large amounts of data as it is less vulnerable to mechanical damage during transportaSon compared with disk.”

Tape Offers the Lowest Cost & Most Reliable Way to Transfer Files

IDC Report. April, 2011. “IDC’s Archival Storage Solu@ons Taxonomy


Oracle’s StorageTek LTFS SoPware Offerings

!  Single Tape Drive Support

!  Available as Open Source SoPware: !  hXp://oss.oracle.com/projects/lZs/

!  Supports StorageTek T10000 and LTO

!  Engineered For: !  Up to 8.5 TB Capacity !  Transport & Distributed Sites !  3rd Party ApplicaSon IntegraSon

Oracle’s StorageTek LTFS, Open EdiLon So[ware

Oracle’s StorageTek LTFS, Library EdiLon So[ware

!  StorageTek Tape Library Support

!  Available on SoPware Delivery Cloud:

!  hXps://edelivery.oracle.com/

!  Supports StorageTek T10000 and LTO

!  User Interface (BUI)

!  Engineered For:

!  5-‐8 Tape Drive Environments

!  Larger Distributed Sites

!  Tape NAS for Entry and Midrange

!  3rd Party ApplicaSon IntegraSon Oracle HSM 6.0

!  LTFS FormaXed Tape Ingest & Export


Oracle HSM 6.0 LTFS Import / Export

1.  Import LTFS tapes into the archive

2.  Export archive data onto LTFS tapes

3.  Read / write LTFS tapes which are not part of SAM archive

–  Similar funcSonality to LTFS-‐LE

New FuncSonality first introduced in SAM-‐QFS 5.4

FILE

FILE INDEX

LTFS Format SAM-QFS Archive

FILE

FILE INDEX

LTFS Format

Export Import

LTFS Access


•  LTFS metadata ingested into QFS file system when a cartridge is placed in library

•  If/when file is requested it is then copied into the SAM archive

• Archived copies are stored in .tar format

• Files exported in LTFS format

LTFS Import / Export


•  Add line to /etc/opt/defaults.conf –  lZs = /mnt_point_to_use

•  This specifies where to mount LTFS media filesystems

–  samd config

•  Import LTFS tape into Oracle HSM –  Insert tape into library, e.g. a T10000 with label LTF001 –  Run samimport –L lib_eq to import tape as LTFS media

–  Use aXach to import LTFS metadata, and ingest to also copy data •  samlZs aXach S.LTF001 /hsm_fs/lZs_001 -‐-‐-‐ or -‐-‐-‐

•  samlZs ingest S.LTF001 /hsm_fs/lZs_001

•  Export LTFS tape label, e.g. S.LTEX01 –  samlZs mklZs S.LTEX01

–  samlZs load S.LTEX01, then copy files to /mnt_point_to_use/LTEX01

Oracle HSM 6.0 LTFS ConfiguraSon and Usage


Periodic Data Integrity ValidaSon


Periodic Data Integrity ValidaSon • T10000C / T10000D and LTO5/6 can verify the integrity of data: – When data is wriXen – When data is read – At any other Sme automaScally

• OpSon to self-‐heal if data errors are found Records have DIV CRC added at host

Records Sent to Tape with DIV CRC

Record DIV CRC checked; write

and read


•  Create /etc/opt/SUNWsamfs/verifyd.cmd –  Enable feature

•  pmv = on

–  E.g. Only do validaSon between 6 PM and 5 AM •  run_Sme = 1800 0500

–  Specify verificaSon type, either: standard (default), complete, complete plus, legacy •  pmv_method = complete

–  Specify equipment to use, either: all (default), library, (opSonal) drives •  pmv_scan = library 200 max_drives 2

–  Specify acSon to take on detected errors, either: none (default), recycle, rearch •  pmv_acSon = recycle 10 // Recycle tape aPer 10 hard errors detected

•  pmv_acSon = rearch age 5m // Rearchive error files aPer 5 minutes

Periodic DIV configuraSon


Ephemeral ID Support


Oracle HSM 6.0 Ephemeral ID Support •  Allows Ephemeral IDs to be used with QFS

–  Solaris Joins AcSve Directory (AD) Domain

–  QFS filesystem shared through CIFS

–  Any authenScated AD user can use QFS share without explicit idenSty map

•  DefiniSons –  Ephemeral ID

•  Temporary Unix ID (UID/GID) >= 2^31

•  Must not be stored in filesystem •  Mapped by idmapd(1M) to SID

–  SID •  Windows Security IdenSfier

•  Domain-‐Idx (S-‐1-‐5-‐12-‐7623811015-‐3361044348-‐030300820-‐1013) •  Must be somehow stored in filesystem


Oracle HSM 6.0 Ephemeral ID Support •  No QFS configuraSon needed for feature to work • When incoming Ephemeral ID detected by QFS

– Mapped to SID

–  Domain & IDX stored in .domain & .fuid files

–  Encoded index into .fuid file stored in filesystem as (0x80000000 | fuid_index) •  Owner (UID) •  Group (GID) •  ACL Entries

• When outgoing FUID detected by QFS – Mapped to SID using .fuid & .domain files

– Mapped to Ephemeral ID

–  Returned to Solaris

announcing: samqfs** rebranding*to:*konferenz-nz.dlr.de/pages/samfs2015/present/1... ·...

Documents

announcing: samqfs** rebrandingto:konferenz-nz.dlr.de/pages/samfs2015/present/1... ·...