Historical shipping data and machine learning Whitaker Institute Research Day NUI Galway 6th April 2017

Aidan Kane Peter M Solar NUI Galway Vesalius College, VUB and Université Saint-Louis-Bruxelles

Note: data/graphs throughout reflect work-in-progress, not final results, and so should not be relied/cited.

Outline

• Database of steamships c.1810 to c.1860 built in Britain and Ireland • 59,000 observations • from c.250 sources • up to 50 fields for each observation

• Database of ships 1770s to 1850 listed on Lloyd’s Register of Shipping • transcription from originals: sample of about 260,000

observations –A,B, ships most years, full sample every 5 years • up to 40 fields for each observation

• How many distinct vessels in each dataset? • => domain knowledge/hand matching and/or machine learning • probabilistic record linkage?

• with INSIGHT, IT at NUI Galway)

Why?

• shipping as a key technology of the industrial revolution • how did technological dimensions evolve?

• how long did vessels stay in service • engine size, propulsion/hull – wood to iron/speed? • in which sectors were vessels adopted?

• impacts on international trade • indicators of regional industrial activity/skills

• how many built, and where? who built them? • a good application of some basic database technologies

• and perhaps more • picks up an interesting strand in econometric tradition of

economic history ...

Journal of the American Statistical Society, 53 (1958), 360-381

sparking a debate ... (TLS (1966), pp. 899, 948, 1016)

Hughes • “His complaint that we missed some ships is likely to be of no quantitative

importance if our 1,945 ships contain no error larger than a good sampling error. If our ships turned out to be as good as a random sample, all our conclusions would hold without modification no matter how many ships we missed.”

• “I rather doubt, from what he says, that he ever read our paper.” • “The question Mr Craig must answer is this: What differences do the missing

numbers make? The question cannot be answered by grousing about the political background, only by hard quantitative work.”

Craig • “I have never seen this phrase (permanent British registry) used before, and I

would ask Professor Hughes which side of the Atlantic it comes from.” • “…econometric historians, now that their work of quantification is rapidly

expanding, should exercise the greatest care in the utilization of statistical material collected in the past.”

On steamships

• Rather than 1,945 steamships built in the period, we think probably closer to 6,000 • wider range of sources • not just one cross section • focus not just on those ships involved in international

trade • also coastal trade, tugs, river/canal craft/postal/military • attempt to trace whole life cycle, including ‘reason out of

service’

When steamships die wrecked, sold Denmark, sold France, driven ashore, converted, sold foreign, sold Russia, sank, lost, grounded, broken up, sold Germany, register still open, sold Australia, sold Spain, sold Portugal, captured, sold Bahamas, sold Gibraltar, destroyed in hurricane, sold Italy, missing, sold Trinidad, hulked, disposed of, boiler explosion, abandoned, sold Finland, sold Switzerland, foundered, sold New Zealand, sold India, sold Poland, scrapped, blown up, sold Brazil, sold Admiralty, sold Turkey, sold Netherlands, sold Naples, sold Greece, sold Norway, sold Moulmein, exploded, stranded, sold Canada, sold Colombia, engine transferred, sold Hong Kong, register closed, inland navigation, last note, sold Egypt, sold Mauritius, sold South America, sold Austria, sold Argentina, sold Japan, sold Uruguay, condemned, burnt, collision, sold East India Co, sold Sardinia, beached, sold Antigua, sold Belgium, sold Africa, sold Malta, sold Tripoli, struck off--owners not found, scuttled, sold Columbia, sold Chile, sold Sicily, sold China, engine removed, sold Burma, sold Sweden, sold Demerara, sold New Brunswick, laid up, withdrawn, sold Ecuador, sold Singapore, sold Mexico, sold Batavia, sold Australian, out of commission, still in existence, disarmed, last observed, ceased, sold Peru, sold Argentine, explosion, sold government, sold Hamburg, defective, failed experiment, sold Prussia, sold USA, sold West Indies, sold Bermuda, still present, sold Quebec, out of registry, destroyed, out of service, sold Jamaica, cancelled, still in operation, out of register, sold Cochin China, closed, chartered France, sold Barbadoes, sold Barbados, sold Venezuela, written off, not steam, sold Asia, sold Vatican, sold to crown, sold Ionia, sold South Africa, under tonnage req, engine removed, condemned as prize, hull for sale, coal depot, broken up, sold Tuscany, deleted, sunk, sold Cuba, sold Ceylon, used as barge, certificate given up, sold Romania, sold Prince Edward Island, sold Gualdalquivir Co, sold Papal States, sold Borneo, converted to barge, sold Paraguay, chartered Turkey, sold Australia, then Batavia, sold Hayti, sold Haiti, still in service, store ship, dismantled, still in use, damaged, sold Belize, sold Honduras, dismasted, sold Thailand, existence in doubt, sold, converted to dredger, sold Trinity House, depot,

What’s in a name? Faid Gahaad, Michael, Cock o`th North, Illawana, St Kieram, Ann Scarboro, Dreadnough, George and June, Jokka, Die Schone Mainaern, Robert and Ann, William and Charles, Pride of the North, Lady Kilburn, Seyd Pacha, Bellbird, Adaline, Candare, Plato, Bonny Dundee, Will-o`-the-Wisp, Sir C Napier, Revensbourne, Cock o`the North, Aquilla, Sun Flower, Forget-me-Note, Malukhoff, Thomas Roydon, Primer Argentino, Koh-i-nor, Normandy, Pellissier, Flying Childer, Earl of Malmsebury, Dune, August Louise, Saint Andrew, Saint Mungo, Storfursten, Chili, General Monanga, Earl of Elgin, Finland, Senator, Silma, Tribune, Lubeck, Earl Douglas, Guide, Lochfyne, Malcolm Brown, Almansor, Correo, Corriere Sicillano, Cotinguiba, Gotherburg, Graaf von Rechteren, Henry Wright, John M`Adam, Nicholai 1st, Santa Cruz, St Halvard, Warata, George the Fifth, Balder, Baron von Humboldt, Eclat, Folden, Havila, Will o` the Wisp, Camoes, Dom Alfonso, Protis, Memel Packet, Der Prusse, Persenunga, Camaragibe, Berlin, Prins Oscar, Queen of the West, Acoriano, Alexandre, Alku, Antonio Varas, Aphroezza, Aracaju, Attalante, Belgique, Boyana, De Cartes, East Anglia, Foz de Douro, General Pelissier, Gipsy King, Glommon, Great Contest, Gustave Pastor, Hadjaz , Hamburgh, Ilmarmen, Iron Master, Istanboul, Italo, Kielmansegge, Killingsworth, Kroonprinces Louise, L`Imperatrice,, Leopold 1, Lisbonne, Marco Bozzaris, Mariout, Marsella, Narwa, Nordstjernan, Oliva, Panelinion, Pelissier, Peninsular, Plexavra, Pytheas, Quarry Maid, Rio de Janeiro, Robert Henry, Sagres, Salama, Salaminia, Sphendone, St Petersburgh, Stromstad, Tilset, Union No 2, Collettis, Comte de Hainaut, Congress, Corra Linn, El Correo del Riff, Eptanissos, Estephania, Fauqui, George Olympius, Guillermina, Marie de Brabant, Restlass, Briliant, Unuique, Christiana, Wiliam Swan, Pauliina, Sea Nymh, Gosport, Belldog, Ocean Pride, Inuique, Wansbeek, Sir William Pool, Admiral Misulis, Tantullan, Cockerill, Izamados, Patrus, Al Hamy Pacha, Colietis, Rendell, Bridge, Hulls, Coventanter, During, Lady Seule, Orrisa, Cockerll, Taumados, Regus Ferreos, Barbudda, Juverne, William & John, Isabelle, Powerfull, Rosaric, Ann

Database coverage

VARIABLE Coverage Augmented

Year built 96 99

Location built 90

Year out of service 71 94

Reason out of service 64

Tonnage 93

Horsepower 86

Builder 78

Engine-maker 50

-400

-300

-200

-100

0

100

200

300

400

500

1810 1820 1830 1840 1850 1860

Flows: ships built and out of service

DISAPPEARANCES

NEW CIVILIAN

NEW MILITARY

NET BUILD

0

50

100

150

200

250

300

350

400

450

500

1810 1815 1820 1825 1830 1835 1840 1845 1850 1855 1860

Steamship Building

Database

Official statistics

Hughes & Reiter

0

10

20

30

40

50

60

70

80

90

100

1810 1820 1830 1840 1850 1860

Diffusion of Iron Hulls

0

10

20

30

40

50

60

70

80

90

100

1815 1820 1825 1830 1835 1840 1845 1850 1855 1860

Regional Diffusion of Iron Hulls

CLYDETYNETHAMESMERSEYOTHER

Lloyd’s Register as a source Easy to search for ships Much harder to search for masters, owners Mass of information on: Ship characteristics Shipbuilding Ownership Uses

Lloyd’s Registers as source

• Fairly comprehensive coverage of ships involved in U.K.’s foreign trade • Including ships owned abroad • But for the coasting trade only a small share of ships,

mainly larger vessels • Relatively few steamships (alternative database)

• Standard format throughout period

MySQL database .

Easy to search for ships Much harder to search for masters, owners Mass of information on: Ship characteristics Shipbuilding Ownership Uses

Lloyd’s Registers as source

• Fairly comprehensive coverage of ships involved in U.K.’s foreign trade

• Including ships owned abroad • But for the coasting trade only a small share

of ships, mainly larger vessels • Relatively few steamships (alternative

database) • Standard format throughout period

MySQL database data transferred from Excel, can be readily interrogated.

Standardising locations

Newfoundland: 58 variants: • N fl'd, N fland, N flnd, N'fdld, N'fdlnd, N'fiand, N'fid, N'find,

N'fl, N'flad, N'fland, N'fld, N'fln, N'flnd, N'fndld, N'fnld, N'wfld, N'wflnd, Newf, Newfd, Newfdl, Newfdld, Newfl, Newfld, Newflnd, Newfndld, Newfoundland, Nfdland, Nfdld, Nfdlnd, Nfdnd, Nfiand, Nfind, Nfl'd, Nfl'nd, Nflad, Nfland, Nfld, Nflnd, Nflndld, Nfndid, Nfndl, Nfndl'd, Nfndld, Nfndlnd, Nfnld, Nufld, Nwfd, Nwfdl, Nwfdld, Nwfl, Nwfl'd, Nwfl'nd, Nwfld, Nwflnd, Nwfndld, Nwfndlnd, Nwfnlnd

So far 6826 distinct abbreviated locations 1135 standard locations

Source potential: one example

• Hull protection: • Doubling: structural solidity • Sheathing: protection against shipworms and fouling

Doubling and sheathing: broad changes (%)

Doubled Sheathed Sheathed Wood Metal

1779 9,8 36,0 0,1 1800 5,5 28,4 13,1 1820 2,8 15,5 24,9 1840 3,3 8,5 32,7 1860 0,8 4,0 45,9

Doubling by location of ships

Baltic White Sea Fishing 1779 18,5 22,4 64,1 1800 10,3 20,0 37,6 1820 4,6 4,7 29,7 1840 1,9 6,3 26,7 1860 1,7 0,0 50,0

Stronger ships well before iron?

Sheathing by destination, 1779-1860

Sheathed

Coasting 10,5

Northern Europe 16,8

Southern Europe 53,1

Americas/Atlantic 72,0

Asia/Africa 91,4

... against the dangers in warmer waters

Next steps

• most steamship/LR data now done • standardisation

• linguistic, geographic similarity/closeness of names, locations especially – others?

• machine learning • for steamships (in a sense, already done)

• cluster analysis • probabilistic record linkage – supervised/with ground-truthed data

• for Lloyd’s register • more open-ended

• match year-to-year a starting point • matches/non-matches/uncertain