toro 1 emu hacking at the peabody museum. yale campus
TRANSCRIPT
![Page 1: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/1.jpg)
Toro 1
EMu Hacking at the Peabody Museum
![Page 2: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/2.jpg)
Yale campus
![Page 3: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/3.jpg)
Peabody CollectionsCounts & Functional Cataloguing Unit
• Anthropology 325,000 Lot• Botany 350,000 Individual• Entomology 1,000,000 Individual• Invertebrate Paleontology 300,000 Lot• Invertebrate Zoology 300,000 Lot• Mineralogy 35,000 Individual• Paleobotany 150,000 Individual• Scientific Instruments 2,000 Individual• Vertebrate Paleontology 125,000 Individual• Vertebrate Zoology 185,000 Lot / Individual
2.7 million database-able units => ~11 million items
![Page 4: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/4.jpg)
Peabody CollectionsFunctional Units Databased
• Anthropology 325,000 90 %• Botany 350,000 1 %• Entomology 1,000,000 1 %• Invertebrate Paleontology 300,000 55 %• Invertebrate Zoology 300,000 20 %• Mineralogy 35,000 85 %• Paleobotany 150,000 60 %• Scientific Instruments 2,000 100 %• Vertebrate Paleontology 125,000 60 %• Vertebrate Zoology 185,000 95 %
940,000 of 2.7 million => 37 % overall
![Page 5: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/5.jpg)
EMu migration in '05(all disciplines went live
simultaneously)
Physical move in '00-'02(primarily neontological disciplines)
Big events
![Page 6: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/6.jpg)
The four YPM buildings
Peabody(YPM)
EnvironmentalScience Center
(ESC)
Geology / Geophysics(KGL)
175 Whitney(Anthropology)
![Page 7: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/7.jpg)
VZKristof Zyskowski (Vert. Zool. - ESC)
Greg Watkins-Colwell(Vert. Zool. - ESC)
![Page 8: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/8.jpg)
HSIShae Trewin
(Scientific Instruments – KGL )
![Page 9: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/9.jpg)
VPMary Ann Turner
(Vert. Paleo. – KGL / YPM)
![Page 10: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/10.jpg)
ANTMaureen DaRos
(Anthro. - YPM / 175 Whitney)
![Page 11: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/11.jpg)
EMu Hacking at Peabody
Hacking – in a laudatory programming sense, not a criminal sense
![Page 12: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/12.jpg)
Mitnick
Often we tend to think of “hackers” in this mode
![Page 13: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/13.jpg)
Mitnick modifiedcracker
A better moniker
![Page 14: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/14.jpg)
Mitnick modified w/EMucracker
Crackers often have unnamed accomplices…
![Page 15: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/15.jpg)
3 Vignettes of YPM EMu “hacks”
• An issue of functionality (background script)
• An issue of performance (tweaking the catalogue)
• An issue of user behavior & cost (another script…)
![Page 16: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/16.jpg)
Hack Vignette #1
Multimedia module - JPEG 2000 support
![Page 17: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/17.jpg)
http://www.jpeg.org/jpeg2000
- non-proprietary compression standard- lossless mode (much smaller files)- lossy mode (vastly smaller files)- potential space/bandwidth savings
![Page 18: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/18.jpg)
http://www.fnordware.com/j2k
![Page 19: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/19.jpg)
JP2 spicebush with J2K and tail target
![Page 20: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/20.jpg)
JP2 spicebush tails with file sizes
1.54 mB (native TIFF) 15 kB (heavily squeezed JP2)
![Page 21: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/21.jpg)
HERBIS images
261 kb – <1% 1,302 kb – 2%
5,166 kb – 12% 62,640 kb – 100%
![Page 22: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/22.jpg)
JP2 – no thumbnailIn EMu, oops… no thumbnail
![Page 23: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/23.jpg)
JP2 – script coding
find imagedir –name *.jp2 –mtime -2 –print
loop on the matches and test to see which recently loaded JP2 files are missing a thumbnail JPG, or which JP2 files have been modified more recently than their existing thumbnail JPG ; then build filenames for any qualifying target JPGs ; execute script several times per hour from cron
jasper –f match –F tempfile
convert tempfile –resize 90x90 target
![Page 24: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/24.jpg)
JP2 – prior, withoutscript wakes up every 20 minutes…
![Page 25: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/25.jpg)
JP2 – now, withmakes the thumbnail…
![Page 26: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/26.jpg)
JP2 – Tiled View
JP2 files now behave just like all other standard multimedia
![Page 27: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/27.jpg)
JP2 – Photoshop opens
Double click and the Photoshop handler kicks in
![Page 28: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/28.jpg)
JP2 – V1
V. 1 – simply generated thumbnails in the background
![Page 29: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/29.jpg)
JP2 – V2
V. 2 – also inserted suitable metadata into records via texload
(next version, script to be called directly in validation code at file time)
![Page 30: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/30.jpg)
Hack Vignette #1
Moral #1 = EMu is extensible, you may be able to implement significant changes yourself in whole or in part, without delay
![Page 31: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/31.jpg)
Catalogue module - performance issues
Hack Vignette #2
![Page 32: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/32.jpg)
Default EMu “cron” job configuration
late night
workday
evening= emulutsrebuild
= emumaintenance batch
= emumaintenance compact
Mo Tu We Th Fr Sa Su
Orange is time EMu busyrunning background jobs.Interfering with workdaywork, and leaving Sundayprocessing time idle/unused.
![Page 33: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/33.jpg)
The ecatalogue database is a rate limiter
File Name Function
~/emu/data/ecatalogue/data the actual data~/emu/data/ecatalogue/rec indexing (part)~/emu/data/ecatalogue/seg indexing (part)
At YPM, the combined size of these was >10 gB, with 4 gB in data and 3 gB in both rec and seg
![Page 34: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/34.jpg)
Touch many types of records in EMu…
e.g., Party record add middle namee.g., Bibliography record add authore.g., Collecting Events recordadd collector
…automatic changes subsequently propogate to numerous records in the ecatalogue database
…ecatalogue can grow a lot and slow EMu to varying degrees between maintenance runs
![Page 35: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/35.jpg)
How to make ecatalogue go faster ?
![Page 36: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/36.jpg)
maybe save 20+% ?
Make it smaller - trim nulls from Legacy Data ?
![Page 37: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/37.jpg)
Repetitive scripting of texexport & texload jobs
Conducting around a million re-imports of records
Manual adjustment of nightly cron jobs to accommodate
Do the work at nighttime over a month-long period
Watched ecatalogue closely to keep from exploding disk
Make it smaller - trim nulls from Legacy Data ?
![Page 38: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/38.jpg)
data
rec
seg
Starting situation at YPM for ecatalogue (gB on y axis)
![Page 39: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/39.jpg)
data
rec
seg
delete nulls from AdmOriginalData
![Page 40: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/40.jpg)
sites – round 2
constant data
lengthy prefixes
… not satisfied with just that… here are some other things to possibly trim!
![Page 41: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/41.jpg)
data
rec
seg
delete nulls from AdmOriginalData
shorten prefix on AdmOriginalData
selectively delete AdmOriginalData
>55 % !
![Page 42: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/42.jpg)
catalogue – round 2data
rec
seg
What ecatalogue AdmOriginalData looks like post scripting
![Page 43: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/43.jpg)
Default EMu “cron” job configuration
late night
workday
evening= emulutsrebuild
= emumaintenance batch
= emumaintenance compact
Mo Tu We Th Fr Sa Su
BEFORE
![Page 44: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/44.jpg)
Modified EMu “cron” job configuration
Mo Tu We Th Fr Sa Su
late night
workday
evening= emulutsrebuild
= emumaintenance batch
= emumaintenance compact
* * *
AFTER
Can now squeeze allmaintenance into wee hoursof night, use Sunday, andfully compact ecatalogueevery other day (asterisks)!
![Page 45: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/45.jpg)
Quick backupAlso, all of YPM EMu can now be squeezed onto a thumbdrive
![Page 46: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/46.jpg)
Hack Vignette #2
Moral #2 = know your data, you can put aspects of EMu on a diet and your computer system is likely to thank you
![Page 47: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/47.jpg)
Hack Vignette #3
EMu sessions - licensing and user behavior
![Page 48: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/48.jpg)
Dreaded email
WARNING! 2 KE EMu user(s) are currently
being denied access because all 10 of your KE EMu licenses
are in use. For license upgrades, please contact [email protected]
Dreaded email for sysadmins
![Page 49: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/49.jpg)
Museum Director: "Go license shopping at KE!"
Systems Admin: "VISA or MasterCard?"
The conversation you dreamof but of course never have…
![Page 50: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/50.jpg)
What do you need ?
• Guaranteed license seat for every potential user ?
• Cover maximal number of expected concurrent users ?
• Minimize expenses by minimizing license seats ?
![Page 51: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/51.jpg)
Jess & Lourdes fight (2)
My turn to log in !
%}&$
Dream on, loser !
#@^*
3rd option is dangerous… if you have this you probably have too few licenses
![Page 52: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/52.jpg)
Even with a moderate number of licenses…
… inactive EMu sessions can and will accumulate
![Page 53: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/53.jpg)
Critical research
VARIANT 1: critical research needed, EMu session put on hold
![Page 54: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/54.jpg)
VARIANT 2: both people and computers crash…
Life intervenesMon cherie IRN
View >Attachments
![Page 55: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/55.jpg)
…enter the EMu Grim Reaper Script
seeks out inactive EMu sessions
![Page 56: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/56.jpg)
reaper – script
codingtexlicstatusps -ef
-Grim Reaper wakes up frequently throughout the day -keeps a running table of statistics about each texserver -compares each texserver against a countdown timer -adjusts timer based on activity since last wake up -if some new activity, resets the countdown timer -if no activity, increments the countdown timer -if countdown timer max is reached, kill the texserver
kill –9 texserver_process_id
![Page 57: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/57.jpg)
Tuning the Emu Grim Reaper Script
Change time between wakeup checks
Change number of wakeup check intervals
Tell reaper to ignore certain users
Amend reaper behavior by time of day
Alter how much inactivity is considered bad
32 regular YPM users, 13 runtime licenses
![Page 58: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/58.jpg)
New sessions started per hour, 0800-1700
25
0
Real data prior two weeks in October 2006
![Page 59: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/59.jpg)
Cumulative new sessions started, 0800-1700
80
0
Real data prior two weeks in October 2006
![Page 60: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/60.jpg)
Active sessions, 0800-1700: three slow days
12
0
2
10
6
8
4
Real data prior two weeks in October 2006
![Page 61: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/61.jpg)
Active sessions, 0800-1700: three fast days
12
0
2
10
6
8
4
Real data prior two weeks in October 2006
![Page 62: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/62.jpg)
Cope on phone
It’s telling me, “Licenses
Exceeded?!”
No more worrries
![Page 63: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/63.jpg)
Hack Vignette #3
MORAL = find a licensing balance, but also consider training your users and EMu system
![Page 64: Toro 1 EMu Hacking at the Peabody Museum. Yale campus](https://reader030.vdocuments.net/reader030/viewer/2022012910/56649ca35503460f949629ec/html5/thumbnails/64.jpg)
Happy Scripting, Happy Campers