data processing architectures

Data Structure and Storage

The modern world has a false sense of superiority because it relies on the mass of knowledge that it can use, but what is important is the extent to

which knowledge is organized and mastered

Goethe, 1810

BytesAbbreviation

Prefix Factor

k kilo 103

M mega 106

G giga 109

T tera 1012

P peta 1015

E exa 1018

Z zetta 1021

Y yotta 1024

Market2012

Digital universe estimated at 2.8 YBDoubling every two years

20205.2 TB per person

Data StructuresThe goal is to minimize disk accessesDisks are relatively slow compared to main memory

Writing a letter compared to a telephone call

Disks are a bottleneckAppropriate data structures can reduce disk accesses

Database access

DisksData stored on tracks on a surfaceA disk drive can have multiple surfaces Rotational delay

Waiting for the physical storage location of the data to appear under the read/write headAround 4 msec for a magnetic diskSet by the manufacturer

Access arm delayMoving the read/write head to the track on which the storage location can be foundAround 9 msec for a magnetic disk

Disks

Minimizing data access times

Rotational delay is fixed by the manufacturerAccess arm delay can be reduced by storing files on

The same trackThe same track on each surface• A cylinder

ClusteringRecords that are often retrieved together should be stored togetherIntra-file clustering

Records within the one file• A sequential file

Inter-file clusteringRecords in different files• A nation and its stocks

Disk managerManages physical I/OSees the disk as a collection of pages

Has a directory of each page on a diskRetrieves, replaces, and manages free pages

File managerManages the storage of filesSees the disk as a collection of stored files

Each file has a unique identifierEach record within a file has a unique record identifier

File manager's tasksCreate a fileDelete a fileRetrieve a record from a fileUpdate a record in a fileAdd a new record to a fileDelete a record from a file

Sequential retrievalConsider a file of 10,000 records each occupying 1 pageQueries that require processing all records will require 10,000 accesses

e.g., Find all items of type 'E'Many disk accesses are wasted if few records meet the condition

IndexingAn index is a small file that has data for one field of a fileIndexes reduce disk accesses

Querying with an indexRead the index into memorySearch the index to find records meeting the conditionAccess only those records containing required dataDisk accesses are substantially reduced when the query involves few records

Maintaining an indexAdding a record requires at least two disk accesses

Update the fileUpdate the index

Trade-offFaster queriesSlower maintenance

Using indexes

Sequential processing of a portion of a file

Find all items with a type code in the range 'E' to 'K'

Direct processingFind all items with a type code of 'E' or 'N'

Existence testingDetermining whether a record meeting the criteria exists without having to retrieve it

Multiple indexesFind red items of type 'C'

Both indexes can be searched to identify records to retrieve

Multiple indexes

Indexes are also called inverted lists

A file of record locations rather than data

Trade-offFaster retrievalSlower maintenance

B-treeA form of inverted listFrequently used for relational systemsBasis of IBM’s VSAM underlying DB2Supports sequential and direct accessingHas two parts

Sequence setIndex set

B-tree

Sequence set is a single level index with pointers to recordsIndex set is a tree-structured index to the sequence set

B+ treeThe combination of index set (the B-tree) and the sequence set is called a B+ treeThe number of data values and pointers for any given node are not restrictedFree space is set aside to permit rapid expansion of a fileTradeoffs

Fast retrieval when pages are packed with data values and pointersSlow updates when pages are packed with data values and pointers

Hash for internal memoryHash maps are available in most programing languages

Also known as lookup tablesA key-value pair

Key ValueAfghanistan 93Albania 355Algeria 213American Samoa 1684… …

International dialing codes

Bit map indexesUses a single bit, rather than multiple bytes, to indicate the specific value of a field

Color can have only three values, so use three bits

Itemcode Color Code Disk address

Red Green Blue A N

1001 0 0 1 0 1 d1

1002 1 0 0 1 0 d2

1003 1 0 0 1 0 d3

1004 0 1 0 1 0 d4

Bit map indexesA bit map index saves space and time compared to a standard index

Itemcode ColorCHAR(8)

CodeCHAR(1)

Disk address

1001 Blue N d11002 Red A d21003 Red A d31004 Green A d4

Join indexes

Speed up joins by creating an index for the primary key and foreign key pairnation index stock index

natcode Disk address

natcode Disk address

UK d1 UK d101USA d2 UK d102

UK d103USA d104USA d105

join indexnationdisk address

stockdisk address

d1 d101d1 d102d1 d103d2 d104d2 d105

Data coding standardsASCIIUNICODE

ASCIIEach alphabetic, numeric, or special character is represented by a 7-bit code128 possible charactersASCII code usually occupies one byte

UNICODEA unique binary code for every character, no matter what the platform, program, or languageCurrently contains 34,168 distinct characters derived from 24 supported language scriptsCovers the principal written languages

UNICODETwo encoding forms

A default 16-bit form A 8-bit form called UTF-8 for ease of use with existing ASCII-based systems

The default encoding of HTML and XMLThe basis of global software

Comma-separated values (CSV)

A text fileRecords separated by line breaks

Typically, all records have the same set of fields in the same sequenceFirst record can be a header

Each record consists of fields separated by some other character or string

Usually a comma or tabStrings usually enclosed in quotes

Can import into and export from MySQL

CSV"shrcode","shrfirm","shrprice","shrqty","shrdiv","shrpe""AR","Abyssinian Ruby",31.82,22010,1.32,13"BE","Burmese Elephant",0.07,154713,0.01,3"BS","Bolivian Sheep",12.75,231678,1.78,11"CS","Canadian Sugar",52.78,4716,2.50,15"FC","Freedonia Copper",27.50,10529,1.84,16"ILZ","Indian Lead & Zinc",37.75,6390,3.00,12"NG","Nigerian Geese",35.00,12323,1.68,10"PT","Patagonian Tea",55.25,12635,2.50,10"ROF","Royal Ostrich Farms",33.75,1234923,3.00,6"SLG","Sri Lankan Gold",50.37,32868,2.68,16

Header

Data

JavaScript object notation (JSON)

A language independent data exchange formatA collection of name/value pairsAn ordered list of valuesParsers available for most common languagesExtensions available to import to and export from MySQL

JSON data typesNumber

Double precision floating-pointString

A sequence of zero or more Unicode characters in double quotes, with backslash escaping of special characters

ObjectArrayNull

Empty

JSON objectAn unordered set of name/value pairs

Separated by :Enclosed in curly braces

JSON arrayAn ordered collection of values

Enclosed in square brackets

JSON{ "shares": [ { "shrcode": "FC", "shrdiv": 1.84, "shrfirm": "Freedonia Copper", "shrpe": 16, "shrprice": 27.5, "shrqty": 10529 }, { "shrcode": "PT", "shrdiv": 2.5, "shrfirm": "Freedonia Copper", "shrpe": 10, "shrprice": 55.25, "shrqty": 12635 } ]}

Array

Object

Value

The evolution of hard drives

A history of the hard drive in pictures

Data storage devicesWhat data storage device will be used for

On-line data• Access speed• Capacity

Back-up files• Security against data loss

Archival data• Long-term storage

Key variables

Data volumeData volatilityAccess speedStorage costMedium reliabilityLegal standing of stored data

Magnetic technologyThe major form of data storageA mature and widely used technologyStrong magnetic fields can erase dataMagnetization decays with time

Hard disk drive (HDD)Sealed, permanently mountedHighly reliableAccess times of 4-10 msecTransfer rates as high as 1,300 Mbytes per secondCapacities in Tbytes

Hard disk drive (HDD)HDD unit shipments and sales revenues are declining, though production (exabytes per year) is growing

A disk storage unit

RAIDRedundant arrays of inexpensive or independent drivesExploits economies of scale of disk manufacturing for the personal computer marketCan also give greater securityIncreases a system's fault toleranceNot a replacement for regular backup

ParityA bit added to the end of a binary code that indicates whether the number of bits in the string with the value one is even or oddParity is used for detecting and correcting errors

Data Number of one bits

Even parity Odd parity

0001100 2 00011000 00011001

Mirroring

MirroringWrite

Identical copies of a file are written to each drive in an array

ReadAlternate pages are read simultaneously from each drivePages put together in memoryAccess time is reduced by approximately the number of disks in the array

Read errorRead required page from another drive

TradeoffsReduced access timeGreater securityMore disk space

Striping

StripingThree drive modelWrite

Half of file to first driveHalf of file to second driveParity bit to third drive

ReadPortions from each drive are put together in memory

Read errorLost bits are reconstructed from third drive’s parity data

TradeoffsIncreased data securityLess storage capacity than mirroringNot as fast as mirroring

RAID levels

All levels, except 0, have common featuresThe operating system sees a set of physical drives as one logical driveData are distributed across physical drivesParity is used for data recovery

RAID levelsLevel 0

Data spread across multiple drivesNo data recovery when a drive fails

Level 1MirroringCritical non-stop applications

Level 3Striping

Level 5A variation of stripingParity data is spread across drivesLess capacity than level 1Higher I/O rates than level 3

RAID 5

Magnetic technology

Removable magnetic diskMagnetic tapeMagnetic tape cartridgeMass storage

Solid StateArrays of memory chips

~30 cents per GbyteMagnetic disk is ~ 5 cents per Gbyte

Prices for SSD are decreasing much faster than HDD pricesFasterLess energyMore reliableHandhelds and laptops

Flash driveSmallRemovableSolid stateUSB connectorUp to 1 Tbytes capacityAround 25 cents per Gbyte for smaller capacity drivesAbout 50 cents per Gbyte for larger capacity drivers

Optical technologyA more recent development than magneticUse a laser for reading and writing dataHigh storage densitiesLow costDirect accessLong storage lifeNot susceptible to head crashes

Digital Versatile Disc (DVD)

DVD drives have transfer rates of around 2.76 M bytes/sec and access times of 150 msec Read-only versions

DVD-Video (movies)DVD-ROM (software)DVD-Audio (songs)

DVD-RRecordable (write once, read many)

DVD-RAMErasable (write many, read many)

Blu-ray DiscCapacity of 25 to 50 Gbytes20 layer version can store 500 GbytesVersions for BD-ROM, BD-R, BD-RE

Storage life

Merit of data storage devices

Device Access speed

Volume Volatility Cost per megabyte

Reliability Legal standing

Solid state *** * *** * ** *

Fixed disk *** *** *** ** ** *

RAID *** *** *** ** *** *

Removable disk

** ** *** ** ** *

Tape * ** * *** ** *

Cartridge ** *** * *** ** *

Mass storage ** *** * *** ** *

SAN *** *** *** ** *** *

Optical-ROM * *** * *** *** ***

Optical-R * *** * *** *** **

Optical-RW * *** ** *** *** *

Data compressionEncoding digital data so it requires less storage space and thus less network bandwidthLossless

File can be restored to original stateLossy

File cannot be restored to original stateUsed for graphics, video, and audio files

Recent developmentsDeclining cost of main-memoryMulticore processorsCan get massive performance improvements for business analyticsSAP HANA is a product of these recent developments

Overcomes the disk bottleneck

Rethinking metricsTraditional

Cost per TBNew

Cost per TB per secondMain memory is roughly 10 times more expensive but 1000 times fasterTotal cost of ownership lower as well

Eliminating disk-based database

Faster and cheaper architectureSome firms will have a need for disk for some years because of database sizeTransition as main memory becomes cheaper

Columnar and row-based data storage

A table can be stored as a series of rows or columnsRow-storage typically good for transactionsColumn-storage typically good for business analyticsIn-memory facilitates either approach

And so can disk

In-memory systemsIn-memory can achieve 5-10 compression ratios

Helps reduce cost of the transitionSQL with some extensionsSAP has showcased a 250TB systems with 250 nodes

Case studyHilti

Key pointsDisk drives are relatively slow compared to main memoryStorage devices vary on several parametersSSD gradually replacing HDDSelect a storage device based on storage and retrieval goalsIn-memory database is a recent and growing development