securing containers on the high seas (owasp belgium … · 2020-05-04 · securing containers on...
TRANSCRIPT
![Page 1: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/1.jpg)
Securing Containers on the High Seas
Jack Mannino @ OWASP BelgiumSeptember 2018
![Page 2: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/2.jpg)
Jack Mannino•CEO at nVisium, since 2009•Former OWASP Northern Virginia chapter leader•Hobbies: Scala, Go and Kubernetes
Who Am I?
![Page 3: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/3.jpg)
Container Security Lifecycle
Design Build Ship Run
![Page 4: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/4.jpg)
Containers are __
![Page 5: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/5.jpg)
Containerized Architecture
https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/
![Page 6: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/6.jpg)
Who Does What Now?
![Page 7: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/7.jpg)
Design
![Page 8: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/8.jpg)
Secure Architecture
ü Orchestration & Management - Control Plane
ü Network Segmentation & Isolationü Encrypted communicationsü Authentication (container & cluster-level)ü Identity Management & Access Controlü Secrets Managementü Logging & Monitoring
![Page 9: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/9.jpg)
• Open Container Initiative (OCI) spec promotes a broader set of container tech (life beyond Docker)
• Isolate containerized resources differently
• Goal is to prevent escaping from the container
• Isolation via Namespaces & Control Groups
• Isolation via Hypervisorhttps://blog.jessfraz.com/post/containers-security-and-echo-chambers/
Picking the Right Container Runtime
![Page 10: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/10.jpg)
We can solve security issues through patterns that lift security out of the container itself. Example – Service Mesh with Istio & Envoy
Leveraging Design Patterns for Security
![Page 11: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/11.jpg)
Build
![Page 12: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/12.jpg)
• Build steps focus on code repositories and container registries
• Run Tests -> Package Apps -> Build Image
• Build first level of security controls into containers
• Orchestration & management systems can override these controls and mutate containers through an extra layer of abstraction
Securing the Build Process
![Page 13: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/13.jpg)
Example: Insecurely Configured Docker Container
![Page 14: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/14.jpg)
• Your resources may be built with external tools, formats, or code
• Terraform (.tf), CloudFormation, Helm/Charts, Brigade, Metaparticle, etc.
• Create reproducible builds to streamline deployments
• Example – Helm/Charts use Go templates
Chart for Jenkins https://github.com/kubernetes/charts/blob/master/stable/jenkins/values.yaml
Other Configuration Formats
![Page 15: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/15.jpg)
• Focus on keeping the attack surface small• Use base images that ship with minimal installed packages and
dependencies• Use version tags vs. image:latest• Use images that support security kernel features (seccomp, apparmor,
SELinux)
$ grep CONFIG_SECCOMP= /boot/config-$(uname -r)$ cat /sys/module/apparmor/parameters/enabled
Base Image Management
![Page 16: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/16.jpg)
• Circa 2003, root privileges were broken into a subset of capabilities.• This feature enables us to reduce the
damage a compromised root account can do.• Docker default profile allows 14 of 40+
capabilities.• Open Container Initiative (OCI) spec restricts
this this even further:• AUDIT_WRITE• KILL• NET_BIND_SERVICE
Restricting Root Capabilities
Docker Default Capabilities• CHOWN• DAC_OVERRIDE• FOWNER• FSETID• KILL• SETGID• SETUID• SETPCAP• NET_BIND_SERVICE• NET_RAW• SYS_CHROOT• MKNOD• AUDIT_WRITE• SETFCAP
![Page 17: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/17.jpg)
• More often than not, your
container does not need root
• Often, we only need a subset of
capabilities
• Limit access to underlying host
resources (network, storage, or IPC)
docker run -d --cap-drop=all --cap-add=net_raw my-image
Example – Ping command requires CAP_NET_RAW
We can drop everything else.
Limiting Privileges
![Page 18: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/18.jpg)
Kernel Hardening
• Restrict the actions a container
can perform
• Seccomp is a linux kernel feature
that allows you to filter dangerous
syscalls
• Docker has a great default profile
to get started
![Page 19: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/19.jpg)
Mandatory Access Control (MAC)
• SELinux and AppArmor allow you to set granular controls on files and network access.
• Limits what a process can access or do
• Logging to identify violations (during testing and production)
• Docker leads the way with its default AppArmor profile
![Page 20: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/20.jpg)
• Vulnerabilities can possibly exist in:• Container configurations• Container packages• Application Code & Libraries
• Solutions:• Clair• Dependency Check• Brigade• Commercial tools
Container Package Management
![Page 21: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/21.jpg)
Ship
![Page 22: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/22.jpg)
• Securely move the container from registry -> runtime environment
• Controlled container promotion and deployment
• Validate the integrity of the container• Validate security pre-conditions
Ship
![Page 23: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/23.jpg)
What Am I Even Shipping?
https://kubernetes.io/blog/2017/11/securing-software-supply-chain-grafeas/
![Page 24: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/24.jpg)
Validating Integrity & Signing Builds
• Ensures integrity of the images and publisher attestation
• Sign to validate pipeline phases
• Example – Docker Content Trust & Notary, GCP’s Binary Authorization
• Consume only trusted content for tagged builds
![Page 25: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/25.jpg)
Validating Security Pre-Conditions
• Allow or deny a container's cluster admission
• Centralized interfaces and validation
• Mutate a container's security before admission
• Example – Kubernetes calls this a PodSecurityPolicy
![Page 26: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/26.jpg)
Run
![Page 27: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/27.jpg)
Typically, containers are managed, scheduled, and scaled through orchestration systems.
Kubernetes, Mesos, Docker Swarm, AWS ECS, etc.
• Cluster/Service authentication• Identity Management & Access
Control• Policy & Constraint Enforcement• Propagation of secrets• Logging & Monitoring
Example – Kubernetes Control Plane
Run
![Page 28: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/28.jpg)
Control Plane Hardening
• The Control Plane manages the cluster’s state and schedules containers.• A privileged attack against a control plane node
or pod can have serious consequences.•Managed services such as Azure AKS, AWS EKS
and Google Cloud Platform’s GKE abstract away the control plane for you.
![Page 29: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/29.jpg)
• Deploy, modify, and kill services• Run commands inside of containers• Kubernetes, Marathon, and Swarm
APIs work similarly• Frequently deployed without
authentication or access control
Management APIs
![Page 30: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/30.jpg)
• Authenticate subjects (users and service accounts) to the cluster
• Authentication occurs at several layers• Authenticating API subjects• Authenticating nodes to the cluster• Authenticating services to each
other
Avoid sharing service accounts across multiple services! Example – K8s JWT Generator
Authentication
![Page 31: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/31.jpg)
K8s - Create a Role
K8s - Bind a Subject to the Role
Authorization & Access Control
• Subjects should only have access to the resources they need
• Limit what a single hostile user or container can achieve)
• Multiple vantage points - to the API, between containers, between control plane components
![Page 32: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/32.jpg)
Logging and Monitoring
• OWASP Top 10 2017 – A10 = Insufficient Logging &
Monitoring
• Container lifecycle is short and unpredictable
• Visibility through telemetry and logs
• Tag and label assets for context and de-duplication
• Focus on visibility at these levels
• Application-level logging
• Container-level logging
• Orchestration/Scheduler logging
• Cloud/Infrastructure logging (services and
systems)
![Page 33: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/33.jpg)
Example - Creating a K8s Audit Policy
•Building an audit policy• API accessible via the
audit.k8s.io group• Metadata – user, timestamp,
verb, resources but no request or response
• Request – request only• RequestResponse – request
and response• None - do not log
![Page 34: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/34.jpg)
• Send security relevant events to a Webhook endpoint• --authorization-webhook-config-file=webhook.config
Webhooks
![Page 35: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/35.jpg)
Secrets Management
• Safely inject secrets into containers at runtime
• Reduced footprint for leaking secrets• Dynamic key generation and rotation
is ideal• Anti-patterns:• Hardcoded• Environment variables
• Limit the scope of subjects that can retrieve secrets
![Page 36: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/36.jpg)
Secrets Management
Dockerdocker run –it –e “DBUSER=dbuser” –e “DBPASSWD=dbpasswd” mydbimage
echo <secret> | docker secret create some-secret
Kuberneteskubectl create secret generic db-user-pw --from-file=./username.txt --from-file=./password.txt
kubectl create –f ./secret.yaml
![Page 37: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/37.jpg)
Nothing is Perfect
![Page 38: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/38.jpg)
Beware of Plain Text Storage
Prior to 1.7, secrets were stored in plain text at-rest
$ ls /etc/foo/usernamepassword
$ cat /etc/foo/username
admin $ cat /etc/foo/password 1f2d1e2e67df
As of v1.7+, k8s can encrypt your secrets in etcd
Not perfect at all, either.
![Page 39: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/39.jpg)
https://blog.openshift.com/vault-integration-using-kubernetes-authentication-method/
Dynamic Loading & Rotation
![Page 40: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/40.jpg)
Example - Retrieve and Mount a Secret
![Page 41: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/41.jpg)
Policy & Constraint Enforcement
• Harden by applying a Security Context at the pod or container level• Mutate the container's configuration as needed
• i.e- overrides a Dockerfile
Setting PodSecurityContext SecurityContext
Allow Privilege Escalation
X
Capabilities X
Privileged X
Read-Only Root Filesystem
X
Run as Non Root X X
Run as User X X
SELinux Options X
FS Group X
Supplemental Groups
X
Example – K8s Pod & Container Security Context
![Page 42: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/42.jpg)
Conclusion
• Secure your container ecosystem and supply chain, not just the runtime
• You probably don't need root – start with minimally privileged containers
• Focus on layered security and strong isolation
• Ensure visibility from a developer's laptop to running in production
![Page 43: Securing Containers on the High Seas (OWASP Belgium … · 2020-05-04 · Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018. ... (container & cluster-level)](https://reader034.vdocuments.net/reader034/viewer/2022042305/5ed0bc636897b23f4e436cec/html5/thumbnails/43.jpg)
Thanks! Keep in Touch
Jack ManninoTwitter @jack_manninoLinkedin - https://www.linkedin.com/in/jackmanninoEmail - [email protected]