www.egi.eu egi-engage recent experiences in operational security: incident prevention and incident...

www.egi.eu

EGI-Engage

www.egi.eu

Recent Experiences in Operational Security:

Incident prevention and incident handling in the EGI and WLCG infrastructure

Dr Linda Cornwall, STFC.

HEPiX Spring 2015

24th March 2015 Linda Cornwall, STFC 1

www.egi.eu

WLCG and EGI

• The (Worldwide) LHC Computing Grid and The European EGI Infrastructure share a lot of the same resources

• Also share Security teams and activities

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford2

www.egi.eu

Contents

• Incident Prevention • Policy definition• Vulnerability handling• Security monitoring

• Incident handling and incidents from the last year

• Evolving the work

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford3

www.egi.eu

Security Incident Prevention

• Far more work goes into preventing incidents than handling them• Security Policy definition• Software Security, especially Software

Vulnerability handling• Security monitoring - monitoring for known

vulnerabilities and insecure configuration

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford4

www.egi.eu

Security Policy Definition

• Security Policy definition is carried out by the EGI Security Policy Group (SPG)• Defines the behaviour expected from NGIs, Sites,

Users and other participants to maintain a beneficial and effective working environment

• Output is various policy documents• Parties read and sign, so that they know and

understand what they should and should not do• List of policy docs at:

https://wiki.egi.eu/wiki/SPG:Documents

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford5

www.egi.eu

Minimizing vulnerabilities in the infrastructure

• Handling vulnerabilities found/reported• Main activity of the EGI Software Vulnerability

Group

• Assessing software for vulnerabilities• Formally and informally

• Preventing new vulnerabilities being introduced • Developer education, awareness

• Considering new software to be used in the infrastructure

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford6

www.egi.eu

Software Vulnerability Handling

• Approved procedure (Under Revision)https://documents.egi.eu/public/ShowDocument?docid=717

• Anyone may report an issue• By e-mail to report-vulnerability@egi.eu• This may be because they have found it in software• Or it may be that it has been announced

• If it has not been announced, SVG contacts the software provider and the software provider investigates (with SVG member, reporter as appropriate)

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford7

www.egi.eu

Relevance and Risk

• The relevance to EGI is considered, and what affect it could have

• Then it is risk assessed, and put in 1 of 4 categories• ‘Critical’, ‘High’, ‘Moderate’ or ‘Low’

• If it has not been fixed, target date set• ‘Critical’ 3 days, ‘High’ 6 weeks, ‘Moderate’ 4

months, ‘Low’ 1 year

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford8

www.egi.eu

Advisory issued

• An advisory is issued when vulnerability is fixed if EGI SVG IS the main handler of vulnerabilities for this software, or software is in EGI UMD regardless of the risk.• E.g. Grid Middleware, tools developed in EGI

and collaborating projects

• If EGI is NOT the main handler, e.g.linux advisory only issued if ‘High’ or ‘Critical’

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford9

www.egi.eu

SVG message – if you find a vulnerability

If it is NOT public knowledge •DO NOT

• Discuss on a mailing list – especially one with an open subscription policy or which is archived publically

• Post information on a web page• Publicise in any way without agreement of SVG

•DO report to SVG via

report-vulnerability@egi.eu

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford10

www.egi.eu

High and Critical Vulnerabilities monitored

• Sites are monitored for ‘High’ and ‘Critical vulnerabilities.

• EGI CSIRT chases sites which are exposing ‘Critical’ vulnerabilities

• Sites may get suspended if they expose critical vulnerabilities and don’t respond

• Respond if asked to by IRTF/CSIRT

• For ‘High’ risk, up to the local NGIs.

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford11

www.egi.eu

Vulnerabilities reported during last year

• 42 new entries in vulnerability tracker (RT)• 12 concerned Grid Middleware - 2 critical

(1related to heartbleed, 1 related to perfsonar/cacti) 4 high

• 16 Linux – 3 critical (heartbleed, Shellshock, Kernel) 5 high

• 4 Cloud enabling – 3 high• 6 VO software – 3 high • Others include 1 high

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford12

www.egi.eu

Changing types of Vulnerabilities

• Until about 1 year ago most vulnerabilities concerned Grid Middleware

• Now more concerning VO specific software• Including Data Protection issues • VOs take it into their head to ‘monitor’ activities

in a way that is traceable back to user

• Cloud specific software• Less knowledge about this

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford13

www.egi.eu

Incident Handling

• Approved Incident handling procedurehttps://documents.egi.eu/public/ShowDocument?docid=710

• Incidents are handed by the Incident Response Task Force.

• Fortunately there are not many • Incident prevention is quite successful

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford14

www.egi.eu

Incidents during last year (8)

• Primecoin mining (Policy violation)• Open Hostkey leaking private information• User cert mis-use• Fed Cloud incident

• Due to bad endorsed VM• UI compromised (4 user IDs compromized)• Shellshock related compromises to Perfsonar nodes

(multiple sites)• Compromise due to port left open• DDoS to some EGI services

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford15

www.egi.eu

Evolving the Security Work

• Evolving the security work is necessary due to e.g.• The EGI federated Cloud

• Changing responsibility model• Changing technology

• Long Tail of Science• Different trust model

Have some H2020 funding for EGI engage to carry out this evolution

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford16

www.egi.eu

Policy documents under revision

• Getting rid of ‘Grid’• Policies apply to all technology and services

• Acceptable use policy• External draft – request for feedback and

comments

https://wiki.egi.eu/wiki/SPG:Drafts:Acceptable_Use_Policy_March_2015

• Security Policy for the endorsement and operation of Virtual Machine images• Especially for Fed Cloud experience

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford17

www.egi.eu

New policy documents

• Data Protection Policy• Formerly only had “Grid Policy on the handling

of User Level Job accounting data• Finding Data protection policy needed as User

level data is being monitored and exposed inappropriately.

• Long Tail of Science Policy• Related to allowing access other than by large

VOs, IGTF certificates• User sub-proxy.

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford18

www.egi.eu

Vulnerability handling evolution

• Now more software is coming into use where SVG members have no knowledge• New members of SVG who know about cloud

software, especially tools written within the community

• ‘Expert’ contact for all software Cloud enabling software deployed in the Fed Cloud

• VO software – assume VO security contact is responsible and know who to contact

• No more than 2 steps to the right person.

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford19

www.egi.eu

Software security checking

• For some community cloud enabling software have a detailed ‘Technology provider’ questionnaire

• For other software propose something simpler:-- • License details• How long will it be under security support?• How are security problems reported? • Are security problems announced? • Check compliance with Data Protection policy• Some other simple technical checks – e.g. is user input

is validated, bad constructs – not obviously bad

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford20

www.egi.eu

Incident response evolution

• Changing responsibility model in the cloud will mean changes to incident response.

• A lot of work is going on including traceability – See Ian Collier’s talk

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford21

www.egi.eu

Questions??

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford22

www.egi.eu24th March 2015

Linda Cornwall, STFC. HEPiX Spring conference, Oxford

23