customer distributed denial of service (ddos) experiences - networkshop44
TRANSCRIPT
Denial of Services (DDoS)
Steve Knibbs
@ulcc www.ulcc.ac.uk
Denial of Services (DDoS)
Steve Knibbs
Head of Infrastructure Services
University of London Computer Centre (ULCC)
@ulcc www.ulcc.ac.uk
DDoS Experiences
What happened and what we learned
@ulcc www.ulcc.ac.uk
Never assume it won’t happen
• 50 years in business without any serious attacks
• First serious attack
in 2015
• Plan now because it will happen to you
@ulcc www.ulcc.ac.uk
Things we got wrong
Powers of Persuasion
@ulcc www.ulcc.ac.uk
Things we got wrong
• Initially attributed to a Firewall
software update
• Our ability to have visibility of
external traffic, relied on firewall
logs/interface which we could
not access
@ulcc www.ulcc.ac.uk
Things we got right
• Communication (Web site, Twitter, email)
• Roles and responsibilities
• Protection of technical staff
• Quick engagement with 3rd party support
@ulcc www.ulcc.ac.uk
What we’ve done since
• External Network Monitoring –
SNORT
• Improved our OOB admin access to
equipment
• Improved our processes (Steps to take,
informing Police, etc)
• Implemented ‘BGP blackhole’
• Implemented further anti-DoS policies
on the firewall
@ulcc www.ulcc.ac.uk
…more improvements
• New Firewall with additional security
features
• Massive improvements to our core
infrastructure
• Improved governance and senior
management awareness
• Some customers moved to a Cloud based
‘washing’ service
• DDoS mitigation services have been
considered but ruled out for the time being
@ulcc www.ulcc.ac.uk
What we’re doing next
• Dedicated security team
@ulcc www.ulcc.ac.uk
Jail Sentence
…sometimes we do catch the bad
guys!
Sentenced to four years and 10
months for carrying out cyber-
attacks and holding a cache of
weapons
@ulcc www.ulcc.ac.uk
Questions
• Thank you for your attention
• Questions
jisc.ac.uk
Steve Knibbs
Head of infrastructure services, University of London
Distributed Denial of Service Attacks (DDoS)
Mike Turpin
Mike Turpin
Head of Network Services, UCL
Distributed Denial of Service Attacks (DDoS)
Timeline
Thursday 12th November 15.50-16.50
Target: Mail & Web
– Blocked ~3000 IPs at MAN router
– JANET blocked UDP 1900 inbound to server
Thursday 12th November 20.00-21.00
Target: Web
– Reflected DNS and UDP fragments sourced from open DNS resolvers
– JANET rate limited those ports to 5Gb/s
Friday 13th November 14.00-15.00
Target: Shibboleth & DNS
– DNS Amplification
– JANET blocked
Friday 13th November 18.00-19.00
Target: CS DNS server
– DNS and UDP fragments
– JANET added ns1.cs to rate limit
Bandwidth
Thursday
Bandwidth
Friday
Attacks sometimes used to hide other bad things!
Cost
• Reputation
• Lost work
• Lost revenue
• Estimate £250k!
Lessons Learnt & Mitigation
• CSIRT & Network Operations teams were invaluable
• Outsourcing websites isn’t a solution (They just get disconnected!)
• Changed Firewall monitoring (Logging added to load)
• Firewall redesign with DC work (separates campus and DC traffic)
• Assessing DDoS mitigation services ( Procurement)
DDoS Mitigation Services
Commercial providers reassuringly expensive?
£8K/Month (+VAT)! ?
Cheaper Options?
*#$%
Ideal Solution
• Lower cost
• Protection on demand (maybe not always enabled)
• Option to exclude selected traffic
• Automated (out of hours?)
• Alerts
Questions?
CSIRT for tirelessly monitoring our traffic for attacks
John Seymour and his team for implementing blocks/rate limits rapidly
And for providing instant updates on the situation
Without the above help we would have had to sit it out!
jisc.ac.uk
Mike Turpin
Head of network services, UCL