matt carroll - "security patching system packages is fun" said no-one ever
TRANSCRIPT
![Page 1: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/1.jpg)
Join the conversation #devseccon
By Matt Carroll
“Security patching system packages is fun!” said no-one ever.
![Page 2: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/2.jpg)
Matt Carroll - Site Reliability [email protected]
@oholiab
“Security patching system packages is fun!”
said no-one ever.
![Page 3: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/3.jpg)
Some security tasks are just a dragSocial engineering is for protagonists too!
Use tooling to minimise pain and maximise motivation
You CAN use technology to help solve people problems
Lol package management.
The Takeaway
![Page 4: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/4.jpg)
WHO?
![Page 5: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/5.jpg)
mattcQualifications:●Worries too much
Who’s this then?
https://www.flickr.com/photos/ajc1/10994593713
![Page 6: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/6.jpg)
Yelp’s MissionConnecting people with great
local businesses.
![Page 7: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/7.jpg)
Yelp StatsAs of Q2 2016
92M 3272%108M
![Page 8: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/8.jpg)
Building our PaaS: PaaSTA!Managing our edgeSupporting deploys and developer workflows
Server/instance maintenanceToolingThe kitchen drawerBackronyming badly
Not just rebooting and saying no.
Operations team
![Page 9: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/9.jpg)
KC Green
![Page 10: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/10.jpg)
heartbleed.com
![Page 11: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/11.jpg)
KC Green
![Page 12: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/12.jpg)
Patching packages is hard and boringThings that are difficult to upgrade in place without downtime
Technical debt and edge casesPackage freezingYelppacksFrankenLucidDockerTrying to do clever things with apt
So what’s the problem?
![Page 13: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/13.jpg)
Doing clever things with apt
http://scarfolk.blogspot.co.uk/2013/05/the-dont-campaign-and-kak-1973.html
![Page 14: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/14.jpg)
#!/bin/bash[ -z "$1" ] && exit 2
library_regex="$@"
mapped_deleted=""while read process; do pid=$(echo $process | awk '{print $1}') cmd=$(echo $process | awk '{print $2}') map=$(sudo grep -E "$library_regex" /proc/$pid/maps 2>/dev/null) if ! [ "$map" = "" ]; then echo -e "\n${process}\n------------" echo "$map" if echo "$map" | grep -q "(deleted)"; then mapped_deleted="$mapped_deleted\n$cmd" fi fidone< <(ps --no-header -eo pid,comm)
if [ "$mapped_deleted" = "" ]; then exit 1else echo echo "NEEDS RESTART" echo -n "=============" echo -e "$mapped_deleted" | sort | uniq exit 0fi
IN B4 APT-GET UPGRADE
![Page 15: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/15.jpg)
For tech debtFor reproducible buildsBecause apt does silly thingsTo generally be aware of impact of CVEs on critical components
At least until we’ve built confidence
But it’s really really dull!
Needs eyes on from an engineer
![Page 16: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/16.jpg)
A JIRA projectUbuntu Security Notice emails straight to tickets
Wheel of Misfortune
Enter AUTOSEC
![Page 17: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/17.jpg)
Engineers tend to like:
Interesting workNew thingsTo know where they standA tight feedback loopAgency
Tedious stuff is tedious
![Page 18: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/18.jpg)
Engineers tend to like:
Interesting workNew thingsFeedbackAgency
Tedious stuff is tedious
https://pixabay.com/p-155981
![Page 19: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/19.jpg)
Engineers tend to like:
Interesting work ❌New things ❌Feedback ❌Agency ❌
Tedious stuff is tedious
![Page 20: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/20.jpg)
Engineers tend to like:
Interesting work ❌New things ❌Feedback ❌Agency ❌
Tedious stuff is tedious
![Page 21: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/21.jpg)
I am not a wizard:
Interesting work ❌New things ❌Feedback ✅Agency ✅
What do I think we can fix?
![Page 22: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/22.jpg)
Aim to reduce MTTR for security tickets within Q2Resolution within 2 weeks unless critical
Close out all pre-Q2 2016 tickets
We had organizational buy in
Already we have a better idea of where we stand as a team.
Enter AUTOSEC OKR
![Page 23: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/23.jpg)
Automate distribution of workIncluding deadlines
Tighten up feedback loop with metrics and frequent reports
Make the critical path to decisive action more explicitImprove documentationMake it easy to get helpImprove perceived agency
How?
![Page 24: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/24.jpg)
Asking non-security specialists to make security decisions
“Won’t Fix” is against engineering naturePrevent naive interventionismYou are making tradeoffs (absolute security vs moving faster than competitors)
You only find out if you did the wrong thingEmpower people to make hard decisions with little payoff
autosec-review mail group (leveraging JIRA again)Anything you can do to make it less painful
Increase Agency: Recognise the futility
![Page 25: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/25.jpg)
JIRA gives us a bunch of stuff for free
We totally have a PaaS to put the Wheel of Misfortune on!(You could totally do this with a cron job)
The AUTOSEC service
Also hooks into JIRA
Work distribution
![Page 26: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/26.jpg)
github.com/Netflix-Skunkworks/go-jira for ad-hoc metrics and mailouts
Helps team members know that they’re helping and what progress on the goals are like
Pretty much the only feedback you get
Feedback: Metrics and reporting
![Page 27: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/27.jpg)
Proactively security patching system packages often feels more like an arcane ritual to satisfy the script kiddie gods than it does engineering. In part, this is because of a feedback loop that’s more of a feedback line… Post completion, you’re safe in the knowledge that you still haven’t been hacked that you’re aware of. Probably.But it’s still important – if your OS vendor has gotten round to announcing and fixing vulnerabilities to you, then they’ve landed in everyone else’s inbox too!This talk will address some of the problems inherent in defensive infrastructure security. It will give examples of how to change the problem space in order to motivate engineers toward being proactive in a field that is “everyone’s responsibility”. Hopefully this should give some insight into how you can leverage technology and pragmatism to instigate change in your security culture. By reducing the pain and uncertainty of taking action, you can make infrastructure security more rewarding and more effective as a result!
Clear up documentation on processEven so, a well defined process on paper is difficult to follow
Did it anywayIncrease agency by REMOVING extraneous information
Break points should happen as early as possibleShould ideally be scripted
Increase Agency: Critical path
![Page 28: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/28.jpg)
Clear up documentation on processEven so, a well defined process on paper is difficult to follow
Did it anywayIncrease agency by REMOVING extraneous information
Break points should happen as early as possibleShould ideally be scripted
Increase Agency: Critical path
![Page 29: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/29.jpg)
Deadlines really help you prioritize work
JIRA and cron(ish) againExtension of AUTOSEC serviceTells you when you’re nearing deadline
“I need it done now/ASAP/yesterday” are not deadlines
Helps to balance against the actually rewarding work
Increase Agency: nagbot
![Page 30: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/30.jpg)
DID IT WORK?
![Page 31: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/31.jpg)
![Page 32: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/32.jpg)
![Page 33: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/33.jpg)
WHAT NEXT?
![Page 34: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/34.jpg)
Those stats are probably liesI hope none of our servers run on Snapdragon kernels…
Scrape information out of USNsAuto-triage information
Introspect with mcollective and what’s in our reposMaybe even auto-close?Feed information to documentation scripts to remove some of the questions
Pointless overhead
![Page 35: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/35.jpg)
When processes are in flux, docs change
I don’t even want to read the docs once, and neither do you
Continue on with scripting workNo extraneous informationProcess is more interactiveProcess changes can be reviewed for greater confidence!
Mental caching
http://www.express.co.uk/finance/crusader/623732/Crusader-act-now-victim-PPI
![Page 36: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/36.jpg)
Remove our old cruft (obviously)
Make puppet dpkg pin versions of packages we install via puppet
apt-get upgrade from upstream security becomes safer
Less complicated process means less can fall through the gaps
Fix packaging
![Page 37: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/37.jpg)
Some security tasks are just a dragSocial engineering is for protagonists too!
Use tooling to minimise pain and maximise motivation
You CAN use technology to help solve people problems
Lol package management.
The Takeaway
![Page 38: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/38.jpg)
Thanks for listening!
![Page 39: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/39.jpg)
@YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp
![Page 40: Matt carroll - "Security patching system packages is fun" said no-one ever](https://reader036.vdocuments.net/reader036/viewer/2022062905/58714e121a28ab55588b7381/html5/thumbnails/40.jpg)
Join the conversation #devseccon
Matt CarrollSRE at Yelp
[email protected]@oholiab
oholiab on Freenode