securing application deployments in multi-tenant ci/cd environments
TRANSCRIPT
Securing Application Deployments inMulti-tenant CI/CD Environments
O’Reilly Security Conference NY 2016
HELLO!I am Binu Ramakrishnan
Principal Security Engineer at Yahoo
You can find me at:@securitysauce
github.com/prbinu
Overview
▹ A brief introduction to CI/CD platform
▹ Challenges with multi-tenant CI/CD environments
▹ Threat modeling
▹ Security patterns and best practices
CI/CD
A CI/CD pipeline is a set of jobs chained to achieve continuous delivery, and deployment of applications to production.
CI/CD DeploymentsSingle-tenant | Multi-tenant
Single-tenant
▹ Dedicated build environments
▹ Distributed
▹ Tenant and provider are the same
▹ Limited security impact
Multi-tenant▹ Shared build environments
▹ Driven by economies of scale
▹ Focused on operational efficiency
▹ Allow enterprise to centrally enforce dev stds & compliance controls
▹ Gateway to enterprise networks
Multi-tenant CI/CD - Security Risks
▹ This platform can be an attractive target for attackers because:
▸ Consolidation of large number of long-lived keys in one place
▸ It acts as a corridor that directly connects to large number of machines, bypassing firewalls
▹ A successful attack can have a significant impact to an enterprise
▸ Leads to exposure of user’s personal info, application credentials, TLS private keys etc.
Denial of Service (DoS) Attacks
* http://bit.ly/2exRpWm
“Enterprises relying on SaaS apps had no choice but to sit and wait and hope their providers got back online as soon as possible.” *
CI/CD in Nutshell
Enterprise CI/CD
Threat Modeling
CI/CD Admin Exclusive access to the CI/CD platform
Actors
User/Developer Person who uses the platform and run their build pipelines.
CI/CD Platform DeveloperLimited admin access to the
platform.
Trust Boundaries
▹ Application - web interface, REST APIs
▹ Host - isolation between host OS and the build job
▹ Network - segment build machines from other machines and from Internet
Entry Points
▹ Build web interface
▹ Commit handlers
▹ Internal endpoints
▸ Build slave endpoint (to launch builds; SSH)
▹ Deployable artifacts
▹ Build notifications - eg. emails, IRC and Hipchat messages
▹ Build console logs
.
Attack Surface
Exit Points
Threats
16
Common Threats
▹ User account compromise & insider threats▸ User and platform admins
▹ Network Intrusion▸ CI/CD internal and external endpoints
Local Storage of Non-ephemeral Keys
▹ Long-lived keys are stored locally to allow build jobs to access protected services.
▹ OAuth tokens, Shared secrets, SSH private keys etc.
Weak Security Controls
Build jobs require access to services as part of their execution (eg. artifacts store). Protection mechanism used by those services are less effective in following cases:
▸ IP whitelist based protection
▸ Shared secrets: shared (and accessible) by all build jobs
The above controls are susceptible to spoofing in multi-tenant environments.
Build Slave CompromiseLarge attack surface, spread across multiple networks (iphone, Android, Server app etc.)
How?▹ A network level compromise, exploiting a vulnerability on build slaves
▹ Jobs break out of build container
Impact▹ Listen to the network, spoof identity and access locally stored
credentials
▹ Gain access to unauthorized machines and steal sensitive data
Building External CodeAn attacker can take this path to get into internal network, either by adding backdoors or exploiting known vulnerabilities with open source software.
Security Patterns for Risk Reduction
Ephemeral Keys
https://www.flickr.com/photos/articnomad/241620406
JWT, OAuth, SSH-CA, AWS access key etc.
▹ Automated deployments require delegation
▹ Token Delegation: a controlled replay of tokens
▹ Ephemeral keys go well with token delegation
model
Capability Delegation & Stateless Auth
https://www.flickr.com/photos/18946008@N06/14551311971/
By passing capabilities (in the form of tokens) with build events, we eliminate the need to store long-lived keys in CI/CD platform
▹ Separate out auth components from the core build platform
▹ The Auth service intercepts and authenticates build events, and issues job tokens to downstream
build platform
▹ Stateless: Build platform no longer stores job specific long-lived keys
▹ Distributed: Per tenant Auth service for cloud based environments, hosted inside tenant’s network
▹ No consolidation of keys in one place
Audit Trails▹ Immutable and append-only
▹ Traceability: logs should capture build pipeline events end to end from commit to final deploy.
▹ Trail traits: who, how, when, sequence of events, start and end state.
▹ Audit trails produced by the Auth service can provide a verifiable chain of trust from commit to deploy.
Container & ToolChain Hardening
Network Segmentation
Colocating CI/CD platform with other corporate machines allow malicious build jobs to raid your internal network.
The recommendation is to zone CI/CD platform from other corporate systems.
Minimal Builds▹ Do not expose PII or other sensitive info to build machines.
▹ Do complex tests outside of build, in a separate environment you
control.
Few more...SSH: Use Restricted ShellsHeadless SSH access for automated deployment should use a restricted shell.
Roll Keys PeriodicallyEstablish a process to periodically roll trust anchor keys (and do it periodically).
Restrict Job Console LogsRestrict build job console logs only to authorized users
Enable 2FAAdmins must follow good security hygiene and use 2FA to access platform application and hosts.
Prune Admin Access ListKeep admin list small for build systems and Git repo access.
Vulnerability Patch MgmtMaintain an inventory of all packages in use and have a mechanism to patch the system in response to a disclosure.
Acknowledgements▹ Christopher Harrell
▹ St John Johnson
▹ Mike Shema
▹ Jeremiah Wuenschel
Let’s recap major conceptsEphemeral KeysReplace long-lived keys with time-bound scoped tokens. Service providers should start supporting tokens for auth & authz.
Capability DelegationDelegate job capabilities with build events, and eliminate the need to store long-lived keys in CI/CD platform.
Audit TrailsA verifiable chain of trust based on traceable audit logs is a foundational requirement for CI/CD.
Minimal BuildsAvoid pulling PII or other production sensitive data to build environments. Keep the builds to minimum.
Network SegmentationNetwork level Isolation of CI/CD machines.
Build ToolChain HardeningBuild tools and Docker containers must be adequately hardened.
THANKS!Any questions?
You can find me at:@securitysaucegithub.com/prbinu
CREDITS
Special thanks to all the people who made and released these awesome resources for free:▹ Presentation template by SlidesCarnival▹ Photographs by Death to the Stock Photo
(license)