Migration: It isn’t just for the Birds

Presented by Becky Bell MPLA/NDLA/SDLA

Tri-ConferenceOctober 5, 2002

Data Migration What is it? Who does it? How can you prepare? When is it done?

Data Mapping MARC Records Non-MARC Records

Migration ConsiderationsData MappingSerials Migration ComponentsData Mapping Detail

MARC records Non-MARC records

Data Cleanup

What is data migration? Simply, moving data existing in one software

application to a different software application Usually is more complex than expected due to

Changed functionalityField size incompatibilityDisparate record structuresExisting/new/no standards supported in either

previous or successor software

Two “Truths” about Data Migration 1. There is always a bigger mess than you

thought. 2. Your assumptions are always wrong.

• From “An ABC guide to data migration,” by Ross Bentley, Computer Weekly, Feb 7, 2002, p.39, and attributed to Richard John, managing director of data migration at Alchemy.

“Data migration not the most fascinating aspect of implementing a new system.”

Data migration is a matter of ABC Analysis of data Building conversion programs Cleaning incorrect data

Remember GIGO--Garbage in, garbage out

Who does data migration? Any entity (library, library consortium,

banks, retail stores, etc.) moving data from one software application to a different application

Focus today is library data Ideally, library staff and vendor work

together to create data mapping specifications

How can you prepare?Read and learn about

• Windows • MARC21 Holdings Format

Know what you currently have• Become familiar with your system’s data dictionary• Document your workflow• Document local practices

Work on data cleanupLearn as much about your new system as possibleTalk to others who have completed their migration

When is it done? Obviously, the switch to production Milestones along the way

Decisions made about what must migrateData mapping specifications writtenSubset(s) converted, loaded, and indexedData reviewed for mapping and data accuracyAs many iterations as needed to get what you

expect

Keep in mind that if data migration is skimped, muffed or even rushed, the success of the new system can be dangerously compromised.

Details matter!

Migration Considerations

What data must be migrated?What data are you willing to re-key?What data are you willing to lose?What data are you willing to share if

you’re part of a consortium?Can uncataloged/unlinked items

exist in your new system?

Migration Considerations

Can bibliographic records exist without items and/or holdings records?

How much training and/or consulting services from the vendor can you afford? Pre-conversion training? Systems administration? Functional training?

Remember that policies and privileges aren’t migrated; you’ll need to create them in new system.

Serials Migration Components

StandardsMARC21 Bibliographic FormatMARC21 Authority FormatMARC21 Holdings FormatNISO Z39.71-1999 Holdings Statements for

Bibliographic Items

No standards forSubscription, claiming, routing dataAcquisitions data (Orders, Vendors, Budgets)

Serials Migration Components

Serials Serial data will be stored in bibliographic,

holdings, item, acquisitions, and serials modulesWon’t look at Patrons, ILL, Booking mapping today

Linkages to maintain or create: serial record to bibliographic and holdings records to items and orders

Serials records not used for check-in can be mapped to a bibliographic record (and can be suppressed from the public catalog)

Data Mapping

Two types of library records MARC

BibliographicAuthorityHoldings

Non-MARCItem

Data mapping

Patrons• Loans• Holds• Cash Transactions• Blocks

VendorsOrders

• Monograph• Serials• Standing orders

Purchase orders (invoices)

Data mapping

BudgetsSerials

• Subscription records• Publication patterns• Claims• Routing lists• Binding records

Interlibrary Loan recordsBooking records

Data mapping detail

MARC bibliographic records If consortium, need to choose system

architecture first (if there are options)One bibliographic/authority/holdings database

for all or separate databases for each library?If one bibliographic database, migrate each

library’s record or merge like records?If records merged, what are the merge criteria?Physical union catalog or virtual union catalog?

Data Mapping Detail

Examples of differences in bibliographic records for Cricket and Library JournalCricket cataloged by Jamestown College

(NDJ) on January 24, 1991 and by Fargo Public Library (NFG) on February 1, 2000

• Differences– NFG has a revision date in 010– NFG has additional subfields in 040 ($$d NST

$$d NSD $$d DLC $$d IUL)– Tags 260, 300, and 321 formatted slightly

differently, probably based on cataloging rules current when cataloged

Data Mapping Detail

– 362 tag formatted pre-Z39.71-1999 Holdings Statements for Bibliographic Items in NDJ and post-standard in NFG

– 901 tag capitalization differs– 936 tag identifies different surrogates used

for cataloging• Additions

– NDJ record has 265 and 350 tags included– NDJ added local holdings information in 500

tag– NFG record has 037 and 130 tags included– NFG record has two 500 tag notes

Data Mapping Detail

• NFG added a 590 tag for local holdings information

• NFG includes 850, 890, and 994 tags

Data Mapping Detail

Library Journal cataloged by Augustana College (SDA) on May 2, 1991 and by Dakota Wesleyan University (SDW) on November 5, 1992

• Differences– In 012, SDA has $$i 8510 and $$m l– In 012, SDW has no $$I and has a $$m c– In 040, SDA has many more $$d’s than SDW– Contents of 265 different—which is more current?– In 321, SDA includes a $$b– Contents of 350 different—which is more current?– Second 650 in SDA record has a $$w

Data Mapping Detail

– 780 in SDA record has a $$w– Contents of 936 are different in SDA and SDW

• Additions– SDA record added a 310, a 362, nine 510

fields• What happens when the newest record overlays

the first record loaded?

Data Mapping Detail

Possible merge criteria Each record converted to new system

Pro--each library retains its cataloging and retains local cataloging policies

Con--lots of duplicates

First bibliographic record into system becomes record for all, based on one common field (bib id number)Pro--cataloging may not be acceptable to allCon--Keeps database size smaller

Data Mapping Detail

First bibliographic record is supplemented as additional records are convertedPro--Additional access points get added for

retrievalCon--Records can become very long Con--False search results become more likely

for your patronsCon--Improper MARC tags add information

more appropriately stored in the holdings record

Data Mapping Detail

Merging algorithm determined, for example, 100, 245 subfield a, 250, 260 subfield b and c match exactly

Pro--De-duplication of database accomplishedPro--All like titles merged into one recordCon--Like titles may be different formats

Sample specifications from PALS to Aleph mapping

No one right way--study options, then choose what provides results you desire

Data Mapping Detail

MARC Holdings records Legacy library systems won’t have holdings

records to convert, so records will be created Specifications need to exist that detail what

data goes into tags and subfieldsNUC symbol, location, call number, and pre- and

post-input stamp information Serials captions Enumeration and chronologyURL stored in 856

Data Mapping Detail

Local notes

Will need to create location/call number mapping documentSample location mapCall number pick listCall number (x) suppression099 call numbers or other non-standard

numbersSuDocs numbers

Sample specifications

Data Mapping Detail

Items Uncataloged/unlinked items will need

bibliographic records Specifications need to include mapping

bibliographic data in items to proper MARC fields for brief records

Links between subscription records and items need to be defined

Data Mapping Detail

Sample itemsSample specificationsSample data map

Data Mapping Detail

Course reserves If serials titles or issues are on reserve,

need to preserve reserve information and status

Sample specificationsSample data map

Data Mapping Detail

Vendors Used by both acquisitions and serials

Acquisitions uses for ordering and payingSerials uses for claiming missing issues

Sample specifications Sample data map

Data Mapping Detail

Orders Orders need to exist for active serials

(check-ins) Sample specifications Sample data map

Data Mapping Detail

Purchase orders (invoices) Payment history Sample specifications Sample data map

Data Mapping Detail

Serials Prediction patterns

Library Journal exampleCricket exampleCoding detail

• Seasons• Free text• Supplements or other additions to regular pattern

Memberships

Data Mapping Detail

Inactive serials records Decision/internal/history/public records

Sample serial recordsSample specificationsSample data map

Data Clean-up

Bibliographic/Holdings records Identify commonly misspelled words in records

in your catalog (use http://www.aallnet.org/sis/tssis/tsll/22-03/obs-locl.htm as a source) and correct

Identify records with duplicate call numbers Identify records with no call numbers Browse your title index for incorrect filing

characters (A, An, The) and correct Begin discussions about indexing

Data Clean-up

Non-MARC records Review reports for obvious errors Weed collections Resolve outstanding overdues, fines and

bills Inventory Review existing notes and remove those

no longer needed Monitor binding lists

Data Clean-up

Purge closed orders and purchase orders (invoices)

Review open orders

Recommendations

Staff development issues Windows MARC21 Holdings Format training Update (or create) procedure manuals to reflect

current practices Document local practices Attend conference programs about standards

and migration issues

Subscribe to open mailing lists for next vendor

Recommendations

Involve all staff in data mapping planning and data review

Develop a good sense of humor--it will be one of your most valuable tools

Remember you have a life outside of data migration!

In Closing,

Review your dataWrite detailed conversion

specificationsReview converted data by comparing

data in old and new systemsDetails matter!