surviving a server hack: lessons learned ... - wordpress… · software only site •millennium,...
TRANSCRIPT
SURVIVING A SERVER HACK: LESSONS
LEARNED & WAYS TO SECURE YOUR
SYSTEM
Lou Balek, Information Security Specialist
Marie Martino, Systems and Catalog Services Librarian
WILIUG SPRING CONFERNCE 2015
MORAINE VALLEY COMMUNITY COLLEGE
• 2 year public cc in SW suburbs
just outside Chicago
• Spring 2015
• FTE: 9,066
• Headcount: 15,293
• Collection:
• 111,000+ Items
• Recently moved from Turnkey
Software Only Site
• Millennium, 2011
SysLib’s Log : Stardate 68421.4
(December 15, 2014)
3:31pm – call from IT about excessive
traffic on server/iii contacted
4:51pm – Millennium is reported
down/iii contacted again
6:55pm – iii contacted with request to
follow-up
H A C K E D!
SysLib’s Log : Stardate 68421.4
(December 15, 2014)
7:53pm - sent email to key staff
with iii update; early morning
meeting scheduled
8:55pm - strategic communication
approved and sent out to public
about the system being down
REBUILDING PROCESS
iii disables Millennium and ability to restart it
IT takes server offline
IT wipes server
IT reinstalls OS & iii reinstalls Millennium
iii restores the DB
IT brings server back online
All of us test
All of us(re)configure
All of us deal with lingering/hidden issues
DB RESTORATION
Bumpy start to process seemed to set the tone for remainder
System was completely down about 43.5 hours
Core public services were fully restored first* (catalog, WAM)
Millennium back end was partially operational after additional 12 hours (early post-reconfig, testing, and fresh backup)
Some components needed to be pieced back together
Took a full month to recover
*Well, mostly, if you don’t count Encore, more on that later . . .
CHALLENGES
Timing Issues
Never a good time for a hack . . .
Finals Week
Just before our winter break (closed
for 2 weeks), but also lowest usage
time
Encore server hardware failure, in the
same week!
CHALLENGES
Preparation
Didn’t have a “disaster” plan
Library had a new relationship with IT
Dept and iii after recent Millennium
server migration
Hindsight is 20/20
CHALLENGES
Restoration is not immediate nor is it
without consequence
Technical hiccups & glitches & snafus (oh
my!) happen and slow down the process
Possibility of data loss
May have to reconfigure a number of system
settings, like starting from scratch
BROKEN “STUFF”
Transactional data loss
Reset all system passwords
Thousands of locked records
Broken OCLC connections
ILL problems:
Requests lost and broken
WEBPAC forms failed/locking
ILL/OCLC connection broken
illoptions form-- for mapping to OCLC form reconfig
Route statements for Encore had to be restored
Diacritics missing
Reconnecting web services (patron sync)
Ongoing issue with backups/enterprise API
WebBridge settings lost
CHALLENGES
Communication/Coordination
Lots of hands working in different
places/different time zones
Logistical issues cropped up during
initial restore and beyond
Project hand-offs were not always
smooth
CHALLENGES
Communication/Coordination (cont.)
Needed PM, iii side! At least flag our account in system!
Ultimately increased amount of time to fully recover system
Defining expectations (urgency/priority)
Emotions running high at all levels, across organization
CHALLENGES
Users
Managing communication with public
Depending on the data you keep about patrons,
may have to address the stolen data issue
Patron trust?
LESSONS LEARNED
Complacency is your worst enemy—have a plan in place!
Communication/coordination with ALL stakeholders is key—
state needs/expectations explicitly.
Advocate for your system, doggedly, if expectations not
being met and reach out to those who can help.
Remember, the restoration process takes time. Be
prepared for “bumps in the road,” possible data loss, and
configuration issues.
Prioritize services--users first!
Yes, it can get worse before it gets better (but it will get
better).
Document the process – learn from previous incidences.
Get to know your Security Specialist!
MILLENNIUM/SIERRA SECURITY TIPS
http://innovativeusers.org/past-iug-conferences.html
IUG 2012 A03: Security 101 for System Administrators, presented by Daniel Ferrer from Central Michigan U and Doug Randall, iii
http://csdirect.iii.com/documentation/presentation_archive.php
IUG 2014 D09:IUG 2012 A03: Security 101 for System Administrators, presented by Chris Pettibon and Doug Randall, iii
ATTACKS ON WEEK
42 Bash OS Vulnerability attack
Over 15 different countries
TOOLS OF THE TRADE
Firewall
Vulnerability scanner
OS / software patches
Logging events
IP blocking
NEXT-GENERATION FIREWALLS
Next-generation firewalls combine
application awareness and deep packet
inspection to give companies more control
over applications while also detecting and
blocking malicious threats.
NESSUS
Vulnerability scanner for auditors and
security analysts.
Patch Auditing
Risk Assessment
OS / SOFTWARE PATCHES
Update OS include features and bug fix
LOGGING EVENT
When an event happen write this log file
Login event
Process event
IP BLOCKING
Black list
By event type
By county
OPEN-SOURCE SOFTWARE
OSS is computer software with its source code
made available with a license in which the
copyright holder provides the rights to study,
change, and distribute the software to anyone
and for any purpose.
OPEN-SOURCE TOOLS
OpenVAS - vulnerability scanner
Snort - lightweight network intrusion detection
system
OSSEC - is an Open Source Host-based
Intrusion Detection System
Graylog2 - log analyzer and searching through
log errors.
ANY QUESTIONS
