Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident reportingGridKa Summer School 2010
Stefan Freitag, Florian Feldhaus
Robotics Research InstituteTU Dortmund
September 10, 2010
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Contents
1 Before you report
2 Incident Scenarios
3 Incident handling
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Do you know....?
Security Incident Response Policy1
objective: ensure that all incidents are investigated as fullyas possible and that sites promptly report intrusions.
As a grid participant, you agree to
report suspected security incidents that have impact orrelationship to grid resources, services, or identitiesrespond to and investigate incident reports regardingresources, services, or identities for which you areresponsibleperform appropriate investigations and forensics and sharethe results with the incident coordinatorfollow the incident response procedure
Next question: what is the incident response procedure?1https://edms.cern.ch/document/428035/7
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
EGEE incident response procedure2
Audience
grid site security officers and site administrators
Definition of security incident
The act of violating an explicit or implied security policy
Definition of actions for the case of a security incident
More on this in a few minutes . . .
2https://edms.cern.ch/document/867454
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Security incident - scenario A (2009)
Some grid sites allow gsissh-based access to VoBoxes (e.g.for VO software managers)
On a VoBox Grid users are mapped to local accounts
Initial step for an attacker
gain access to user credentials (certificate or proxy)
What happens next ?
Connect to VoBox using stolen credentials
Running e.g. a kernel exploit to gain root privileges
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Security incident - scenario A (2009)
# sh −x wunderbar empor ium . sh[ . . . ][+] got r i n g 0 ![+] d e t e c t ed 2 .6 s t y l e 4k s t a c k s[+] D i s ab l e d s e c u r i t y o f : noth ing , what an
i n s e c u r e machine ![+] Got r oo t !
sh−3.00# idu id=0( r oo t ) g i d=0( r oo t ) g roups =64004( hepcg )
con t e x t=u s e r u : s y s t em r : i n i t r c t
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Security incident - scenario B (2010)
Department A The Grid
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Security incident - scenario B (2010)
Department A The Grid
CERTIFICATEX.509
CERTIFICATEX.509
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Security incident - scenario B (2010)
Department A The Grid
CERTIFICATEX.509
CERTIFICATEX.509
Alienattacker
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Security incident - scenario B (2010)
The Grid
CERTIFICATEX.509
Alienattacker
stolen
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
For the next slides please keep in mind:
The red block describes actions required by the EGEE IncidentResponse Procedure document
The blue block contains information about actions carried outduring a security incident at the Grid resource in Dortmund
Down here you will find additional information, e.g. max.response times
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
First action
Inform immediately your local security team and your ROCSecurity Contact
Action
Sent E-Mail to Ursula Epting
Read Incident response procedure
Informed 2nd site security officer and local security team
max. 4 hours or
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
Response procedure
In case no support is shortly available [...] try to contain theincident. For instance by unplugging the network cableconnected to the host. Do NOT reboot or power off the host.
Action
Disconnected affected workernodes from network
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
Response procedure
Assist your local security team and your ROC Security Contactto confirm and investigate the incident. Announce the incidentto all the sites.
Actions
Send a heads-up e-mail (template: next slide)
Arranged meeting with local security team
Network guys were asked to check logs
max. 4 hours (Announcement)
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Heads-up E-mail
** PLEASE DO NOT REDISTRIBUTE ** EGEE-<DATE> (ex: EGEE-20090531)
** This message is sent to the EGEE CSIRTs and must NOT be publicly archived **
Dear CSIRTs,
It seems a security incident has been detected at <your site>.
Summary of the information available so far:
Ex: A malicious SSH connection was detected from XXXXX. The extent of the
incident is unclear for now, and more information will be published in the coming
hours as forensics are progressing at our site. However, all sites should check for
successful SSH connection from XXXXX as a precautionary measure.
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
Response procedure
Report a downtime for the affected hosts on the GOCDB
→ Send an EGEE broadcast announcing the downtime forthe affected hosts Use ”Security operations in progress” asthe reason with no additional detail both for the broadcastand the GOCDB.
Actions
Created downtime for possibly affected hosts udo-ce01/udo-dcache01
max. 1 day after discovery
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
Response procedure
Perform appropriate forensics and take necessary actions toprevent further damage
Identify and kill suspicious process(es) as appropriate, butaim at preserving the information they could havegenerated
If it is suspected that some grid credentials have beenabused or compromised, you MUST ensure the relevantaccounts become suspended
If it is suspected that some grid credentials have beenabused, you MUST ensure that the relevant VOmanager(s) have been informed.
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
Response procedure
Perform appropriate forensics and take necessary actions toprevent further damage
If it is suspected that some grid credentials have beencompromised, you MUST ensure that the relevantcertification authority gets informed.
If needed, seek for help from your local security team orfrom your ROC Security Contact
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
Action
Banned affected users on our compute elements by addingtheir DN to the blacklist in/opt/glite/etc/lcas/ban users.db
E-Mail to VO manager regarding compromised user
Contacted the certification authority
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
Response procedure
As part of the security incident resolution process, sites areexpected to report the following information:
affected hosts and hosts used as entry point to the site
remote IP address(es) of the attacker
evidence of the compromise, including timestamps
what was lost, details of the attack
list of other sites possibly affected (if available)
possible vulnerabilities exploited by the attacker (ifavailable)
actions taken to resolve the incident
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
Response procedure
Tracked down the UI that was used by the attacker for jobsubmission (checking logs of batchsystem, ComputeElement, . . . )
Analyzed netflow to/fro affected workernode
Analyzed executables deployed by the attacker
Updated incident report regularly
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
Response procedure
Coordinate with your local security team and your ROCSecurity Contact to send an incident closure report includinglessons learnt and measures taken to prevent future incidents.
Actions
Preparation and submission of final report
max. 1 months
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Incident handling
Response procedure
Restore the service, and if needed, send an EGEE broadcast,update the GOCDB, service documentation and procedures toprevent recurrence as necessary
Actions
Re-installation of affected workernode
Safety tuning
Incidentreporting
S. Freitag, F.Feldhaus
Before youreport
IncidentScenarios
Incidenthandling
Thanks for your attention!