How the Crowd Outperforms Traditional Security Testing
Sr. Security Engineer@leifdreizler
Your Elastic Security Team.
So What Does Bugcrowd Actually Do?• Incorporate up to 17,000 freelance security
researchers as part of a public or private engagement
• Run a crowd sourced pen test • Manage an ongoing bug bounty program
What’s a bug bounty program?
A Brief History of Bug Bounty Programs
These and other companies trust Bugcrowd
Things We’ll Cover
• How to incorporate Crowdsourced Security into DevOpsSec • Accelerating your RO(security)I • What’s in it for me (as a security person)? • Bug bounty fun facts, pitfalls, and war stories
introduce crowd sourcing
Bug Bounty Programs Responsible Disclosure
Crowdsourced Penetration Test
…because people are the new automation
[REDACTED] eCommerce provider
• Long time customer of [EXPENSIVE WEB APP SCANNER] getting “clean results”
• A Researcher gained super admin access through a chained attack within 24 hours of launch
• They thought they were doing a great job at writing secure code…
assume it’s broken
Instructure received 5-10x the number of unique vulnerabilities compared to previous pen tests
Case Study (Company A)
• Gone through previous security testing and remediation with a reputable webapp pentesting vendor
• Expecting low priority results
• 6 P1s
• 4 P2s
• ~30 P3/P4
Case Study (Company B)
• Building a new application
• Had internal security testing built into the SDLC
• 5 P1s
• 16 P2s
• ~30 P3/P4
Lots of bugs == great dev training
Software is always going to have bugs
[REDACTED] Financial Services
• Extortion attempt from Eastern Europe
• Resolved by creating a “one man bug bounty” (we didn’t tell him he was the only one though…)
• Bug received in 15 mins
History
0
125
250
375
500
1995 2000 2005 2010 2015
Adoption of bug bounty and vulnerability disclosure programs.
Bug bounties are awesome…
Minimize Investment
Maximize Quality
Accelerate RO(security)I
Makes a Statement
It’s not just about being cost-effective,
or loud…
It’s about leveling the playing field…
…but bug bounties are hard.
Plan ahead
The mistake *everyone* makes:
VULNERABILITY DATA
PEOPLE
[REDACTED] Digital Advertising
• Engaged Bugcrowd to help them assess the state of the code
• So many valid vulnerabilities submitted they shut down the bounty in 24 hours
• Thrilled with the results!
The Golden Rule:
Touch the code ==
Pay the bug
Align expectations before you engage
Bug bounties create controlled incidents…
[REDACTED] Online Marketplace
• The DevOps and Security teams watched vulns being submitted in real time
• Non-security minded people learned a lot from the process
• Great insight into how ‘good guys that think like bad guys’ work
Mozilla
Thanks to @mwcoates http://www.slideshare.net/michael_coates/bug-bounty-programs-for-the-web
Clearing their assurance debt
Boogeymanbelief
DevOpsSec feeling confident?
Try a Gamified Pentest
1. Create a pool that benefits your engineering team (team drinks, party, event, whatever)
2. Replace an existing pentest w/ a time-boxed bug bounty program
3. Pay out from the reward pool
4. What ever the hackers don’t get, DevOpsSec gets to keep.
Great things happen when you tighten the security feedback loop between your engineers, and what they consider to be
the outside world
Bugcrowd Stats• 28% US based, 28% based in India
• 90 countries have contributed
• Great Britain has low submission numbers, but high average priority
• 37k Total Submissions/6.3k Valid and Unique (17%)
• 16% of Valid Submissions are P1 or P2
• 54% of Paid Programs have at least one P1 or P2
• 93% of those Programs have 2+
• 18% XSS, 10% Logic Flaws, 9% CSRF, 6% Info Disclosure, 2% SQLi
Content Security Policy
Content Security Policy
• Designed to prevent XSS attacks
• unsafe-inline, unsafe-eval, script-src
• report-uri, and report-only mode
• http://c0nrad.io/blog/csp.html
• https://blog.matatall.com/
Highlights from the 2014 Facebook Report• Started in 2011
• Currently $500 minimum, no defined maximum
• 17,011 Submissions
• 61 Eligible bugs were high severity
• 123 Countries (65 Rewarded)
• $1.3 million paid to 321 researchers
Countries with High # of Valid SubsValid Bugs Average $
RewardIndia 196 $1,343Egypt 81 $1,220USA 61 $2,470UK 28 $2,768
Philippines 27 $1,093src: https://www.facebook.com/notes/facebook-bug-bounty/2014-highlights-bounties-get-better-than-ever/1026610350686524
Highlights from the 2014 Github Report
• First year of the program
• $200 - $5,000 (recently doubled upper end)
• 1,920 Submissions
• 73 Unique Vulnerabilities (57 medium/high)
• 33 Unique Researchers earned a total of $50,100 for the med/high vulnerabilities
src: https://github.com/blog/1951-github-security-bug-bounty-program-turns-one
Highlights from the 2014 Google Report
• Started in 2010
• Paid over 200 researchers over $1.5 mil
• $150k highest single payout
• Over 500 unique and valid bugs
• Over half of the bugs in Chrome were reported and fixed in beta or dev builds
src: http://googleonlinesecurity.blogspot.com/2015/01/security-reward-programs-year-in-review.html
Looking Forward with Microsoft in 2015• Started in 2013
• Recently added Azure and raised max payout for “Online Services Bounty Program” to 15k
• Added Project Spartan
• “Mitigation Bypass” bounty and “Bonus bounty for Defense” focus on novel methods to bypass active mitigations (e.g. ASLR and DEP)
• Pay up to $100k for exploit + $50k for defense
src: http://blogs.technet.com/b/msrc/archive/2015/04/22/microsoft-bounty-programs-expansion-azure-and-project-spartan.aspx
Conclusion• Bug bounties are cost effective, and highly marketable, but that’s not
the full story…
• …they create controlled incidents that can powerfully impact the security awareness of your builders.
• Allow people that have historically been ‘builders’ to see how ‘breakers’ think
• Get DevOps to believe in and defeat the boogeyman
The premier platform for crowdsourced security testing.
We’re hiring!