ibm ltfs fktg 2013
DESCRIPTION
IBM LTFS für ME& FTKGTRANSCRIPT
© 2012 IBM Corporation© 2013 IBM Corporation
IT Tape IBM LTFS – Tape File System
IBM Data Protection&Retention
Storedata
Josef Weingand Leading Technical Sales Professional
+49 171 5526783 - [email protected]
Infos / Find me on: http://sepp4backup.blogspot.de/ https://www.xing.com/profile/Josef_Weingandhttp://www.linkedin.com/pub/josef-weingand/2/788/300http://www.facebook.com/josef.weingand http://de.slideshare.net/JosefWeingandhttps://www.xing.com/net/ibmdataprotection
© 2013 IBM Corporation
Source: If applicable, describe source origin
Agenda
Tape Historie und Entwicklung
IBM Tape Entwicklung und IBM Tape Produkte
IBM Tape File System / LTFS
© 2013 IBM Corporation
Video Tapes
1956 Quadruplex format (Zweizoll)
1960 Helical Scan
© 2013 IBM Corporation
1952: IBM Modell 726 erster Bandspeicher• Lineare Aufzeichnung• 1.44 MB• 7-Spur Technik
720 Meter Bandlänge 100 BPI
Photo 1951 Prototyp Photo 1952 IBM 726 in Betrieb
1952-
201260 Jahre
IBM Bandspeicher
© 2013 IBM Corporation
Tape Technologie Entwicklung
Kapazitätssteigerung:–2002: 1 TB Demonstration
• (44h@50Mbps)• Verfügbar 2008 mit TS1130
–2006: 8 TB Demonstration • (355 h@50Mbps)• Verfügbar ~ 2012 – 2015
–2010: 35 TB Demonstration • (1555h@50Mbps)
–Weitere Forschung an 100 TB Tape Medium–Plan: 135 TB Demonstration
Funktionalität–2010: Tape File System (LTFS) für Single Laufwerk–2011: LTFS für Libraries–2012: LTFS Storage Manager
© 2013 IBM Corporation
Tape Storage: Demonstrating 29.5 Gb/in2
Record storage density on magnetic tape of 29.5Gbit/in2
– 39x increase over LTO 4 drives– A bit is 49 nm wide and 446 nm high– Made possible thanks to advanced
BaFe media, improved signal processing, and enhanced mechanics
Strawman Operating Point for 100 Gbit/in2
– Linear Density = 800 kbit/in – Track Density = 125 ktrack/in– Bit area = 32 nm × 200 nm– Only 36 magnetic particles per bit!
© 2013 IBM Corporation
Storage Bit Cells and Extendability
Scaled Bit Cells
Magnified View of Scaled Bit Cells
TAPE8000 nm x 65 nm1.2 Gbit/in²
HDD74 nm x 13.5 nm635 Gbit/in²
NAND45 nm x 45 nm330Gbit/in²
Patterned Media25 nm x 25 nm1000 Gbit/in²
© 2013 IBM Corporation
http://gmailblog.blogspot.com/2011/02/gmail-back-soon-for-everyone.html…. I know what some of you are thinking: how could this happen if we have multiple copies of your data, in multiple data centers? Well, in some rare instances software bugs can affect several copies of the data. That’s what happened here. Some copies of mail were deleted, and we’ve been hard at work over the last 30 hours getting it back for the people affected by this issue.
To protect your information from these unusual bugs, we also back it up to tape. Since the tapes are offline, they’re protected from such software bugs. But restoring data from them also takes longer than transferring your requests to another data center, which is why it’s taken us hours to get the email back instead of milliseconds.
So what caused this problem? We released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email. When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version.
Feb 2011: Google restore gmail from
tape
Hard Lessons Learned
Tape is „last line of defense“!
© 2013 IBM Corporation
Tape ist ein integraler Bestandteil einer effizienten Storage Lösung
Tape stellt eine zusätzliche und andere Schicht gegen Datenverlust zur Verfügung – „offline Medium“
– Langzeit, geringe Kosten, geringe TCO, einfache Skalierung
Reduziert die Kosten– Preis pro TB ist ~10% von Tier 2 disk1
– Stromverbrauch ist ~1%, verglichen zu Disk2
Bietet hohe Kapazitäten und schnelle Transferraten
Bietet hohe Datensicherheit– Geringe Bit-Error-Rate– Read-after-Write Verification
Transportierbar– Leicht, kompakt, hohe Kapazität
Bietet Investmentschutz mit skalierbaren und kompatible Systemen
Über 80%
betrachten Tape als integralen Bestandteil des Backup ProcessesSource: Enterprise Strategy Group Research Report, 2010 Data Protection Trends, April 2010
Sources: 1. “Top 10 Strategies for Surviving Unconstrained Data Growth,” Gartner Symposium Presentation, October 2010, slide 212 “In Search of the Long Term Archiving Solution – Tape Delivers Significant TCO Advantage over Disk”, The Clipper Group, Inc., December 2010.
© 2013 IBM Corporation
New!
Tape DrivesTape Libraries
Tape Virtualization & D2D
LTO 4• 800 GB• 120 MB/sec
TS1130• 1 000 GB• 160 MB/sec
TS7700• VTS für zOS
TS7650G• DeDup Gateway• Cluster• bis zu 3 200 MB/sec
TS3500• 58 – 20 087 Slots•Shuttle: 15x20 087 = 300 000 Slots
TS3310• 30 - 408 Slots
TS31/3200• 22 / 44 Slots
TS2900• 9 Slots
1 HH
2/4 HH1/2 FH
18 FH
192
Media
LTO 6• 2500 GB• 160 MB/sec
TS7620• SMB Appliance• bis zu 145 MB/s• 6 / 12 / 23 / 35 TB native
B
D2D•DS3500, V3700, V7000, XIV, DS8000
IBM Tivoli Storage Manager IBM FlashCopy Manager TSM for VE IBM FastBack
TS1140• 4 TB• 250 MB/sec
Neu!
IBM ReadVerify Appliance (RVA)
IBM Tape File System/LTFS LTFS Storage Manager
TSLM
Neu!
Neu!
LTO 5• 1500 GB• 140 MB/sec
IBM DP&R Storage / DataProtection Overview
© 2013 IBM Corporation
LTO – Über 10 Jahre und mehr…
Oktober
2012
2000 2002 2004 2007 2010 2012
© 2013 IBM Corporation
IBM LTO Generation 6
Schneller und größer zum besseren Peis/Leistungsverhältnis
– 2,5 TB native Kapazität -> ca 110 h @ 50Mbps– 6,25 TB mit Kompression (2,5:1)– 160 MB/s native Durchsatz -> ca 25fache Echtzeit Geschwindigkeit
• Bei 2,5:1 Kompression 400 MB/sec – max 745 MB/sec• Mit variabler Geschwindigkeit 40 – 160 MB/sec
- 14 Speed Matching Geschwindigkeiten- Kein Start/Stop Problem, da kleinste Geschwindigkeit wie bei LTO4
und LTO5• 1024 MB Data Buffer
– Bessere Kompression 2,5 : 1
– Dual Ported 8 Gbit FC oder 6 Gbit SAS
– Kompatibilität: LTO5 Lesen und Schreiben, LTO4 lesen
– Geringerer Stromverbrauch
– Tape File System (LTFS)
© 2013 IBM Corporation
3592 Generation 4 – TS1140 4 000 GB Kapazität (JC) -> ca 177 h @ 50Mbps
– Media Re-Use mit JB• JB: 700 GB -> 1 TB -> 1,6 TB
250 MB/sec native Datenrate Dual-active 8 Gbit FC
– Bis zu 700 MB/sec Durchsatz Neuer Giant Magnetorestive R/W Head mit Aluminium Überzug
– 32 parallel Tracks (Read/Write)
Verbesserter Durchsatz durch– 1024 MB Pufferspeicher– "Virtuelles Back Hitch„ mit noch besserern Leistung– Speed Matching mit 14 Geschwindigkeiten
Sehr schneller Datenzugriff durch – High Resolution Tape Directory– schnellste Spulgeschwindigkeit mit 12,4 m/Sek.– Read Ahead Feature– Schneller Load-/Unload-Mechanismus– JC Media unwesentlich länger als JB Media
Verbesserte End-to-End Datenintegrität– End-to-End CRC Check und „outboarded“ Audit-Funktionen– Für Langzeitarchive nützlich!
LTFS Support als Single Drive und in der Library– Tape wie USB-Stick– Tape wie Disk als File System sichtbar– Sehr großes (52 PB) File System
Upgrademöglichkeit von TS1130 auf TS1140!
Schnellstes Tapedrive im Markt!
© 2013 IBM Corporation Any statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
LTO Generations LTO-3 LTO-4 LTO-5 LTO-6* LTO-7 LTO-8
Max Native Capacity
400 GB (L3) 800 GB (L4) 1.5 TB (L5) 2.5 TB (L6) Up to 6.4 TB (L7)
Up to 12.8 TB (L8)
Other Native Capacities
200 GB L2100 GB L1 R/O
400 GB L3200 GB L2 R/O
800 GB L4400 GB L3 R/O
1.5 TB L5800 GB L4 R/O
2.5 TB L61.5 TB L5 R/O
6.4 TB L72.5 TB L6 R/O
Native Data Rate 80 MB/s 120 MB/s 140 MB/s 160 MB/s Up to 315 MB/s
Up to 472 MB/s* Data Compression engine enhancement from 2:1 to 2.5:1
TS1100Generations
Gen-13592 J1A
Gen-2TS1120
Gen-3TS1130
Gen-4TS1140
Gen-5 Gen-6
Max Native Capacity
300 GB (JA) 700 GB (JB) 1.0 TB (JB) 4.0 TB (JC) 8-10 TB (JD)
14-20 TB (JD)
Other Native Capacities with Media Reuse
500 GB JA 640 GB JA 1.6 TB JB640 GB JA R/O
6-8 TB JC 6-8 TB JC
Native Data Rate 40 MB/s 100 MB/s 160 MB/s 250 MB/s Up to 360 MB/s
Up to 540 MB/s
IBM Tape Drive Roadmaps
© 2013 IBM Corporation17
IBM TS3500 (3584) UltraScalable Tape Library
Hochleistungs - Bandarchiv– Schnellstes Robotorsystem weltweit
Modulares skalierbares Design– Von 58 bis 20 087 Kassetten– bis zu 6 260 TB native Kapazität (derzeit)– Unterbrechungsfreie Erweiterungen
Hochverfügbarkeits Features– Dual Accessor – HA (optional)– Dual Gripper (standard)– Redundant Library Kontroller– Redundant Kontrollpfade
Laufwerke – LTO Gen1, Gen2, Gen3, Gen4, Gen5 und Gen6– TS11x0 Gen1, Gen2, Gen3 und Gen4
Geringster Stromverbrauch von nur (durchs.)185 W
Bis zu 1320 Kassetten pro Frame (0,9 m²)
Bis zu 4,4 PB pro m² -> ca 200 000 h @ 50Mbps
© 2013 IBM Corporation
Warum IT Tapes für M&E
Formatunabhängig
Größere Kapazität
Kostengünstiger
© 2013 IBM Corporation
Tape Entwicklung: Linear Tape File System (LTFS)=> Tape File System
Tape wie „USB-Stick“ – Tape wird als Filesystem dargestellt– Für Single-Drive und Library
• LTO5, LTO6 und TS1140• TS3500, TS31/3200, TS3310 und TS2900
Phase 1: Single Drive Edition– IBM Entwicklung
• Open Source
– LTFS mit einzelnem Laufwerk– Einsatzmöglichkeiten
• Datenaustausch• „Video“ Recorder• Etc
Phase 2: Library Edition– IBM unique– Tape Library stellt sich als File System dar
Phase 3: Software Integration– LTFS Storage Manager
© 2013 IBM Corporation
LTFS: Partitionierung für selbstbeschreibende Medien
Partitionierte Medien sind notwendig für LTFS
Partitionierung der Medien ist die logische und physische Teilung des Bandes in Segmente
2 Partitionen– Index Partition (Partition 0): Enthält die Filesystem Info (ca. 37.5 GB)– Data Partition (Partition 1): Enthält die Files (ca.1.5 TB)
Der LTFS Treiber partitioniert das Medium
Verschlüsselung (Encryption) ist unterstützt– Ein Key für die gesamte Cartridge in transparenter Verschlüsselung– Standard Operation für die das Band nutzende Anwendung
BOT
EOT
Index Partition
Data Partition
Guard WrapsLTFS Index XML
File File File
File
© 2013 IBM Corporation
Recognition by M&E Industry
Association for Information and Image Management (AIIM) Best Practice Award, for the media asset production and management system developed for Shanghai Media & Entertainment Group.
A member of our industry leadership team was recognized in Consulting Magazine’s “Top 25 Most Influential Consulting Leaders of 2005”
The Advanced Media Workflow Association is working with IBM on future development and direction of the open-standards software connectors based on IBM's ASD for media services.
The IBM Linear Tape File System won Broadcast Engineering’s Pick Hit Award at NAB 2011. LTFS puts a file system on top of tape.
IBM Beta Program Customer FOR-A won Broadcast Engineering’s Pick Hit Award at NAB 2010 for their LTO-5 video deck with IBM Long Term File System: LTR-100HS.
Frost and Sullivan named IBM the Global Digital Media Company of
the Year in 2010
Storage Visions
2012 Visionary Media and Entertainment Company Award
2011 Primetime Emmy Engineering Award
IBC Connected World Award
2011
Hollywood Post Alliance Engineering Excellence
Award
© 2013 IBM Corporation
LTFS Single Drive Edition Downloads 2011 - 2012
2011 2012 (thru August)
Total
Linux 953 1,025 1,978
Mac 355 246 601
Windows 1,787 5,571 7,358
Total 3,095 6,842 9,937
LTFS SDE Download Statistics
0
200
400
600
800
1000
1200
1400
1600
1800
Feb Mar Apr May J un J ul Aug Sep Oct Nov Dec J an Feb Mar Apr May J un J ul Aug
2011 2012
WindowsMacLinux
HBO and/or Discovery Channel announce they will accept only LTFS submissionsHBO and/or Discovery Channel announce they will accept only LTFS submissions
“NAB effect”“NAB effect”
© 2013 IBM Corporation
LTFS Library Support
LTFS zeigt Filesystem - Directory der kompletten Library– Jede Kassette wird als separates Directory / Ordner angezeigt – Directroy / Filesystem – Informationen werden von LTFS gecached– Das komplette LTFS Filessystem ist sichtbar und nutzbar (Search) ohne laden von Tape
Kassetten– LTFS steuert Tape Library und Tape Drives
• Lädt Kassetten bei Read/Write Anfroderungen
– „True File System“ Support• Jede Kassette und deren Inhalt ist verfügbar für jede Anwendung durch Standard Filesystem
View of Folders of Cartridges
© 2013 IBM Corporation
Alle Cartridges werden als Directories mit Barcode-Namen angezeigt
LTO-5 Tape Cartridges in Library Magazine
LTFS Library appears as network drive
Windows 7 Explorer
Phase 2: LTFS-LE (Library Edition)
© 2013 IBM Corporation25
FilesCartridge 1
Library
Cartridge 2
© 2013 IBM Corporation
LTFS Vorteile
Tape ist “self-contained”, “self-describing” und portierbar–File Index ist auf Tape
• Hierarchical directory structure• Extended attributes
Tape ermöglicht Cross-Platform Support–Entwickelt für Linux/Mac/Windows
Tape kann nun von jeder Applikation benützt werden –File System (LTFS) wird durch standard OS Interfaces supportet
• Double-click files• Drag-and-Drop• etc.
IBM won
Emmy award
2011!
© 2013 IBM Corporation
LTFS Platform Support
Single Drive Edition Library Edition
Supported Tape Systems
TS2250, TS2260, TS2350, TS2360
TS2900, TS3100, TS3200, TS3310, TS3500, TS1140w/TS3500
Supported Operating Systems
RHEL 5.5, 5.6, 5.7, 5.8, 6.1 and 6.2
SLES 11 SP1 and SP2
Windows 7 SP1
Windows Server2008 R2 SP1
Mac OS X 10.6, 10.7
RHEL 5.5, 5.6, 5.7, 5.8, 6.1 and 6.2
SLES 11 SP1 and SP2
Windows Server 2008 R2 SP1
Other components
For details on supported hardware configurations
please refer to the System Storage Interoperation
Center (SSIC)
For details on supported hardware configurations please refer to the System Storage Interoperation
Center (SSIC)
© 2013 IBM Corporation
LTFS Ecosystem
© 2013 IBM Corporation
IBM Storage Infrastructure
Tape StorageDisk Storage
GPFS / SoFS TSM
CMS MaterialPool
ArchiveCommercials
MAMMaxdome
PlayoutAutomation
FactualArchive
IBM Archive and Essence Manager
IBM Arema & IBM LTFS Storage Manager
LTFS LE
© 2013 IBM Corporation
IBM Storage: LTFS Storage Manager V.1.1 Archive Solution für Digital Media
(Vidoe&Audio)– IBM LTFS Storage Manager managed (archiving
/ restore) Files (Media: Video, Audio, etc) auf LTFS Tape Libraries
Reduziert Kosten in traditionellen Digital Media Umgebungen
Verwaltet Index der Daten– Erlaubt Tagging von Files zum besseren Search
des Content
Open Interface für Enterprise Digital Media Applikationen
– Erlaubt leichte Integration für M&E industry Formats
LTFS Storage Manager für alle Kunden die Media Daten (Audio+Video) speichern und auf Tape archivieren wollen
Disk Cache FC
Manual operationMonitoring
LTFS Library Edition
Local File System
LTFS Storage ManagerLTFS Storage
Manager Agents DB2
FTP, Samba,
NFSLTO Library
Pool 1
Pool 2
Drives
Tapes
Easy of use für Digital Media Daten!
© 2013 IBM Corporation 33
Video Acquisition, Distribution, & Archive
FOR-A Model:LTR-100HSLTO-5 Video Archiving Recorder
•The LTR-100HS enables you to build handy video “archive at ingest” solutions•Can be used as material program exchange media server•Supporting any stage of Ingest - Delivery - Archive - Reuse cycle
http://www.for-a.com/products/ltr100hs/ltr100hs.html
http://www.1beyond.com/products/p2wrangler.asp?search=p2workflow
1 Beyond™ Wrangler™2D & 3D Portable Tapeless Workflow - Ingest, Edit, Archive
•Rapidly unload P2, SxS, RED, Phantom, SI-2K memory cards or disks while shooting in-field •Auto protected to LTO-5 LTFS tape•Instantly Review or Add Metadata to video clips •Optional Editing in-field
LTO-5 Tape
LTO-5 Tape
© 2013 IBM Corporation
Use case - Video Acquisition, Distribution, & Archive
Direct Broadcast
LTFS LE Archive
© 2013 IBM Corporation
Video Archiving in the CloudT3 Media Inc.
Challenges• Needed a low cost delivery platform for enterprise scale Video Supply Chain
as a Service • Information growth of ~100 TB per month• Easy self-serve access required by clients
Solution• IBM Linear Tape File System at several global locations, including some client
facilities • IBM System Storage® TS3200 Tape Library, LTO®-5 tape drives
Benefits• Opened up new business opportunities• Enabled more predictable and transparent pricing for clients• Portable, interoperable, scalable, cost-effective data protection and
long-term storage
‘LTO 5 and LTFS significantly reduce the ancillary costs around storage. This is a real game-changer from IBM’
Mark Lemmons
CTO, T3 Media
TSP03327-USEN-00
T3 with LTFS on Youtube: http://www.youtube.com/watch?v=M7w0jrkQnj4T3 with LTFS on Youtube: http://www.youtube.com/watch?v=M7w0jrkQnj4
© 2013 IBM Corporation
LTFS - Infos
IBM Linear Tape File System:
http://www-03.ibm.com/systems/storage/tape/ltfs/index.html
LTFS Info Center:
http://publib.boulder.ibm.com/infocenter/ltfs/cust/index.jsp
http://publib.boulder.ibm.com/infocenter/ltfsle/cust/index.jsp
http://pic.dhe.ibm.com/infocenter/ltfssm/cust/index.jsp
LTFS Format Specification:
http://www.ibm.com/support/docview.wss?rs=577&uid=ssg1S7003166
LTFS, LFV and ITDT Download (Linux, Mac OS X, Windows):
http://www.ibm.com/support/fixcentral/
(Storage Systems / Tape Systems / Tape Dev Driver and Software / Linear Tape File System / )(Storage Systems / Tape Systems / Tape Dev Driver and Software / LTFS Format Verifier / )(Storage Systems / Tape Systems / Tape Dev Driver and Software / Tape Diagnostic Tool / )
Almaden Research Project:
http://www.almaden.ibm.com/storagesystems/projects/ltfs/
Wikipedia:
http://en.wikipedia.org/wiki/LTFS
© 2013 IBM Corporation 38
Infos / Find me on: http://sepp4backup.blogspot.de/https://www.xing.com/profile/Josef_Weingandhttp://www.linkedin.com/pub/josef-weingand/2/788/300http://www.facebook.com/josef.weingandhttps://www.xing.com/net/ibmdataprotection
© 2013 IBM Corporation 39
© 2013 IBM Corporation
Disclaimers
The performance data contained herein was obtained in a controlled environment based on the use of specific data. Actual results that may be obtained in other operating environments may vary significantly. These values do not constitute a guarantee of performance.
Product data is accurate as of initial publication and is subject to change without notice.
No part of this presentation may be reproduced or transmitted in any form without written permission from IBM Corporation.
References in this document to IBM products, programs, or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM program product in this document is not intended to state or imply that only IBM's program product may be used. Any functionally equivalent program may be used instead.
The information provided in this document has not been submitted to any formal IBM test and is distributed "As Is" basis without any warranty either express or implied. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into their operating environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.
© 2013 IBM Corporation
Trademarks
The following terms are trademarks or registered trademarks of the IBM Corporation in either the United States, other countries or both.
IBM, S/390, ES/3090, ES/9000, AS/400, RS/6000, MVS/ESA, OS/390, VM/ESA, VSE, TPF, OS/2, OS/400, AIX, DFSMS/MVS, DFSMS/VM, ADSTAR Distributed Storage Manager, DFSMSdfp, DFSMSdss, DFSMShsm, DFSMSrmm, FICON, ESCON, Magstar, Seascape
Other company, product, and service names mentioned may be trademarks or registered trademarks of their respective companies.
Windows NT is a registered trademark of Microsoft Corporation.