Varrow

vSphere Security

Jason NashData Center PrincipalVCDX #49, vExpert

Very common question, especially as organizations start virtualizing high profile or tier 1 applicationsThe first step in understanding that question is to look at how virtualized infrastructure differs from physical infrastructure–Faster provisioning of new servers–Many networking configurations move to the host–Hypervisor and the separation of virtual machines–Running multiple servers on a single physical host

Understanding these differences will greatly help you know how managing security differs in a virtual environment

Is Virtualization Secure?

How can rapid provisioning of new severs and infrastructure be a security problem?Often we see virtual environments grow very rapidly without sufficient planningHow vSphere hosts and clusters are deployed can significantly impact how systems must be securedConfiguration settings can be changed much quicker and easier than in a physical environmentProper configuration management and change control must be in place for proper security policy management

Is Virtualization Secure?

Networking in a virtual environment, as we will discuss, is the usual target of security concernsArea of largest concern when it comes to misconfigurationAlso the area of most discussion when planning for different security zones, such as DMZ or other public facing systemsIn many cases the network team no longer manages the network configuration of the vSphere hostAgain, change control!

Is Virtualization Secure?

Virtual machines are “containerized”, is that good or bad?–Allows for virtualization and consolidation–Also makes it very easy to pick up and steal a

complete server just by copying a group of filesThe sprawl of virtual machines in an environment sometimes leads to stale, unpatched, and forgotten systemsVMs can make configuration management more challenging–Virtual Machines can roam–Usually no direct relationship–Very dynamic environments

Is Virtualization Secure?

Isn’t the whole point of virtualization to consolidate servers on to fewer physical hosts? Why is this a security problem?The concern is misconfigurationVery easy to do in a virtual environment and one incorrect checkbox can open up a whole in the security armorStress again proper configuration management and change control

Is Virtualization Secure?

This is a very common question and one that gives security departments nightmares when it shouldn’tAt this time no exploit has ever been seen in “the wild” that would allow an exploited guest to break out of its containerRight now the discussion around busting out of the hypervisor is purely theoretical

Is the Hypervisor a Security Weakness?

In most cases, encapsulating a virtual machine probably helps with securing the systemFar fewer hardware drivers to worry about and manage–Smaller attack surface

Starting to see more security products and technologies that can be plugged in to the hypervisor outside of the VM itself–Less change of being disabled

Downside is that all of the virtual machine is usually stored in a set of files that can be stolen

Encapsulation: Help or Hurt Security?

While virtualization isn’t a new technology for most IT professionals, it can still be daunting for other groups such as those that handle information securityMany myths still persists:–Complete loss of visibility when servers are

virtualized–Hypervisor attacks

The facts are more boring, but need to be communicated:–Solid history of very secure systems–VMware offers many design guides, hardening

papers, and resources for solid, secure architectures–Many virtualized environments with sensitive data or

in hostile positions–Fundamentals of security still apply, and still work

Common Worries About Virtualization Security

So with those things in mind, what do we need to worry about?Oddly enough, it’s very similar to the non-virtualized worldCommon threats to security in a virtual world include:–Proper network access and security, especially

concerning management– Incorrectly configured management–Systems not kept up to date with patches–Overreaching privileges, often due to convenience

Many of these are easy to resolve and avoid with simple planning, management, and monitoring

Types of Security Threats

With respect to security and virtualization, how much is the same and how much is really different?Most things are the same–Systems must still be protected–Guest operating systems must be patched and

protected against malware–Perimeter security is basically the same–Configuration management and change control

should still be followed

Impact of Virtualization on Security

Most of the things that are different are related to the virtualization systems and management–New points of management, such as vCenter, must

be secured–The communication channels between vSphere hosts

and management should be considered–Additional patching of the hypervisor

Depending on how the pre-virtualization environment was architected there may be other areas to protect and secure as well

Impact of Virtualization on Security

Often storage for physical servers is local and with virtualization it is sharedIn virtual environments networks are often collapsed in to single physical connectionsA denial of service attack on one VM may impact other VMs on the host or in the clusterMisconfiguration becomes a much larger problemRoles and responsibilities begin to blur, especially amongst storage and network teams

Impact of Virtualization on Security

VMware has put a lot of focus on security, both in products, technologies, and processesHave made acquisitions to fill gaps in their offerings, such as the acquisition of Blue Lane that turned in to the vShield offeringThey follow a secure software development lifecycle and continually evaluate their products against standards and certifications–List of certifications here: http://

www.vmware.com/support/support-resources/certifications.html

Have a very formalized security response policy should a bug or exploit be found.– Information on the policy is available at http://

www.vmware.com/support/policies/security_response.html

What is VMware Doing About Security?

Continue to build partnerships with third-party partners, such as TrendMicro and othersNew technologies added to the product line, such as VMsafeMore information available at http://www.vmware.com/technical-resources/security/index.html

What is VMware Doing About Security?

COMPONENTS IN VSPHERE SECURITY

There area a lot of parts with virtualization and how to protect them can get very confusingWhile existing tools from the physical world may work, they often are not the most optimal answer–Backups have moved to vStorage APIs–Anti-malware now moving to VMsafe architectures–Network security using new offerings from VMware

and CiscoVMware themselves offers several products and technologies–vShield Zones–vShield App–vShield Endpoint

So many pieces! What do I need to protect what?

That’s a Lot of Pieces. Where Do They Fit?Asset to Protect Feature Technology

VM Backup Backup application supporting vStorage APIs

VM Network Access vShield Zones, Cisco Nexus 1000v, VSG, vASA

Application Specific Network Security vShield App, Cisco VSG and now vASA

Anti-Malware vShield Endpoint and VMsafe

In-depth Network Security & Multi-tenancy

Cisco Virtual Security Gateway and vASA

vSphere Host Access ESXi Lockdown Mode

Storage Access Proper zoning and vendor tools

Log Management Syslog and other third party tools such as Splunk

Host Based Intrusion Detection vShield Endpoint and VMsafe

Hosts and Guests Updates Update Manager and 3rd Party

The vCenter Server and the vSphere hosts can use a combination of usernames for authenticationAllows for a very flexible combinationThe vCenter Server pulls users and groups from:–Windows Active Directory–Local users and groups on the Windows Server where

installedvSphere hosts can allow login accounts from:–A local list created and maintained locally on the

vSphere host–Windows Active Directory, if joined to the domain

Primer on vSphere Authentication

vSphere and vCenter both use user accounts for accessCan be confusing to new administratorsLocal vSphere accounts only exist on that serverUseful for service accounts and other direct accessBecoming less useful as the world moves to ESXiSeveral default users:– root has full administrative privileges– vpxuser which is used by the vCenter Server– dcui user that is used to configure lockdown mode. Do not

modify!

Who Has Access to Your Environment?

Both the vCenter Server system as well as the vSphere host can integrate with Windows AD for authenticationUseful as you can use existing users and groups for roles and permissionsSingle accounts can access the management, virtualization, and other hosts as neededRecommended to use groups for granting permissions and roles instead of individual usersDo not use special Windows groups, such as Everyone, to vCenter rolesThe vCenter Server host can be added to the domain just like any other Windows servervSphere host takes extra steps

Integration with Active Directory

Each vSphere host has its own firewallDon’t confuse this firewall with others such as the vShield tools or Cisco’s Virtual Security GatewayIs used to protect the host and managementNow with ESXi the firewall is almost a forgotten entity…

Fence Off Those vSphere Hosts! The VI Firewall

What’s wrong with using the old methods for protecting VMs against viruses and other malware?VMsafe provides a method to use the hypervisor instead of individual agentsSave a great deal of resources by centralizing this protectionOne shipping product is Trend Micro’s Deep Security

Integrating Security in with the Hypervisor by Using VMsafe

Original technology was acquired with Blue LanevShield has turned in to a suite of products and functions–vShield Zones – VM-aware firewall–vShield App – Upgrade to Zones, more application

intelligence–vShield Endpoint – Used for anti-malware protection

in the guest operating systems

Using vShield to Secure Applications and Guests

VMware provides a central tool that allows you to update both guests (removed in VI5) and vSphere hostsCan also be used for updating the VMware Tools that gets installed in to most guestsAlso will install other agents as needed, such as the Cisco Nexus 1000v Virtual Ethernet Module or PowerPath/VE

Keeping Hosts and Guests Updated with Update Manager

PROTECTING VCENTER

Remember our AAA protocol?–Authentication–Authorization–Accounting

Authentication is handled via the user login–Windows account and password

Authorization is provided via the user roles and permissions–Only allow permissions as needed

Accounting information stored in log files

The Three A’s of vCenter

The vCenter Server system needs to be protected just like any other hostPay attention to vulnerabilities and weaknesses in the underlying Windows operating systemBasic security restrictions at an Enterprise level should include:–Keep the system updated with all needed patches–Only allow vSphere admins to login–Perform standard Windows system protection

• Anti-virus–Limit the number of users that can access the

system–Only give the allowed users the permissions they

require for their work

Best Practices for Deploying vCenter

During deployment of vCenter pay special attention to the user accounts used for services and database accessRecommendations for Enterprise security level–Use a service account instead of the System Account

Word of caution, if an install of vCenter fails delete the install log as it contains sensitive information–Name is hs_err_pidXXX (Where XXX is the process

number)

Best Practices for Deploying vCenter

Recommendations for DMZ level deployments– In secure environments it’s suggested to use a user

account for database authentication instead of Integrated Windows Authentication–Once vCenter is installed remove any unnecessary

rights and permissions that were required–May also want to remove Administrative rights from

the local Windows Administrator account• Assign the rights to a local vSphere admin

account

Best Practices for Deploying vCenter

Recommendations for SSLF (specialized security limited functionality) level deployments–Do not use the vSphere Administrator account for

other regular administrative functions–Create specific accounts for those actions

Best Practices for Deploying vCenter

Limit the network connectivity of the vCenter system to reduce the number of ways an attacker can gain accessIn normal production environments avoid putting it on networks other than a protected management networkDMZ environments require more consideration–Restrict access to the vCenter system–Suggested to use a hardware or Windows firewall–Only allow access to required ports–Disable the managed object browser via web

Best Practices for Protecting vCenter

SSLF environments go even further–Disable the vSphere Web Access–Suggested to disable the datastore browser

To disable the Managed Object Browser set the following element in the vpxd.cfg file on the vCenter Server –<enableDebugBrowse>false<enableDebugBrowse/>

To disable the vSphere Web Access please see VMware KB 1009420To disable the datastore browser enforce the restriction in vCenter permissions

Best Practices for Protecting vCenter

vCenter Server runs as an application on top of Windows–Don’t forget to harden and protect the underlying

Windows operating systemHighly recommended to use the built-in Windows software firewallDon’t forget patches!Also recommended to use a network firewall for added protection–Network firewall configuration may be complex as

vCenter requires a lot of connectivity–Current requirements available at VMware KB article

1022256–Another helpful article is VMware KB 1012382

Hardening the Underlying the Operating System

Linux client components do not perform certification validation!–So even if you have generated certificates they will

not be validated–Vulnerable to man in the middle attacks

The affected client components are–Any vCLI command–Any vSphere SDK for Perl script–VM console access from a Linux web browser–Anything written using the vSphere SDK

Don’t forget the vSphere Client!

Do not allow the Linux components to communicate across an untrusted networkRecommended to use firewalls, jump boxes, and management network restrictionsvCenter Server has a vSphere Client extensibility framework that lets you extend the client menus and toolbars–Only install authorized and trusted extensions

Don’t forget the vSphere Client!

Need to keep an eye on the vCenter Server log files!Stored in–Windows 2003 - C:\Documents and Settings\All Users\Application Data\VMware\VirtualCenter\logs

–Windows 2008 - C:\ProgramData\VMware\VMware VirtualCenter\Logs

Log files are named vpxd-xx-logSample log entry– [2011-06-09 21:47:14.818 02484 warning 'ProxySvc'] SSL Handshake failed for stream TCPStreamWin32(socket=TCP(fd=3192) local=192.168.200.202:443, peer=192.168.200.202:53675), error=SSL Exception: error:14094416:SSL

Also pay attention to the Windows event logsMay want to use a tool such as Splunk for log management

Monitoring the vCenter Logs

VIRTUAL NETWORKING

Before we dive in to the specifics of virtual networking let’s talk designsCommon question! How do we handle different trust zones?–Get this question all the time…

Placing a physical server in a protected environment (production, for example) and unprotected environment (DMZ) would be crazy, but what about vSphere hosts and guests?Why don’t we just replicate how we do it in the physical world?–Greatly lower consolidation ratios–Can’t take advantage of other operational benefits,

such as vMotion and DRS

Deployment Types for Different Trust Zones

Let’s take a look at examplesAssume a company has different trust zonesExamples include:–Production–DMZ (most common)–PCI Compliance zone–Medical Records

There are three usual options for separating trust zones–Partially collapsed but with separate physical trust

zones–Partially collapsed with virtual separation of trust zones–Fully collapsed trust zones

The one you use depends on your security policy, organization, and other legal and regulatory compliance

Deployment Types for Different Trust Zones

Partially Collapsed with Separate Physical Trust Zones

Pros / Advantages Cons / Disadvantages

Very similar to common physical deployments

Lower consolidation of servers due to split clusters/environments

Less chance of a misconfiguration causing a problem

Cost of infrastructure is higher as more equipment is usually required

Less consolidation of duties from current roles

Operational costs are higher as well for the same reason

Support knowledge is less effected Do not get the benefits of a larger pool of resources

Less efficient use of features such as vMotion, DRS, and HA

Partially Collapsed with Separate Physical Trust Zones

Partially Collapsed with Separate Virtual Trust Zones

Partially Collapsed with Separate Virtual Trust Zones

Pros / Advantages Cons / Disadvantages

Greater use and utilization of resources Configuration is more complex due to virtual separation

Can take better advantage of other features such as DRS and HA

Greater chance of a misconfiguration causing a security violation

Reduced number of hosts means less costs

Requires tighter change control and configuration management

Less operational cost due to fewer hosts Suggested regular audits for configuration compliance

Fully Collapsed Trust Zones

Fully Collapsed Trust Zones

Pros / Advantages Cons / Disadvantages

Complete utilization of all resources as needed (no silos)

Greatest complexity of any configuration

Cheapest option of the three due to total collapse of infrastructure

Highest risk of misconfiguration causing a break in security

Entire environment managed from a single place

Virtual appliances used for security enforcement could be misconfigured causing outages or other problemsNetwork configuration complexity could be a problem for support staff

Some things to keep in mind when securing vNetworking1. Keep things simple2. Make sure port-group names are correct and

descriptive3. Duplicate port-groups add confusion4. If you haven’t already, implement VLANs5. Only trunk the VLANs that you need6. Don’t forget to secure those physical uplink switches7. Make sure you know the security requirements of your

VMs and applications8. Understand your legal and regulatory compliance

requirements9. Understand how the layered protection and defenses

work amongst the vSphere products10. Audit your configuration and settings regularly

Top 10 Common Mistakes and Recommendations

The standard vSwitch is easy to configureNo additional components to install or manage

Security Considerations with the Standard vSphere vSwitch

Standard security options– Promiscuous Mode

• Allows a VM to see traffic from other VMs• Useful if you need to run a packet sniffer or traffic

analyzer• Recommended to be Reject

–MAC Address Changes• Allows a VM to change its MAC address• Some configurations, such as load balancers and

clustering, may require this to be set to Accept– Forged Transmits

• Allows a VM to send a frame with a different MAC address than the one assigned

• Use cases are similar to those for MAC Address Changes

Security Considerations with the Standard vSphere vSwitch

Also very easy to configureNo additional components to install or manageSame standard security options–Promiscuous Mode –MAC Address Changes –Forged Transmits

Main considerations are around availability–vCenter is key–Attacker could perform a denial of service attack

Security Considerations with the vSphere dvSwitch

Configuration and implementation of the Cisco Nexus 1000v is far more complex than that of the standard switchesIt also requires other components to be installed, managed, and securedBut for the complexity you do gain other security features–Access Control Lists for VMs and port-groups–Separation of network duties from vCenter

management–More granular Quality of Service controls–Security policies migrate with VMs unlike physical

switch policiesAlong with these you can also add other products in with the Nexus 1000v, such as the Cisco Virtual Security Gateway product

Layering Additional Functionality with the Cisco Nexus 1000v

Why do we bother with isolating management communications?–Direct access to many management interfaces–Not all traffic containing sensitive data is encrypted,

like vMotion and FT– Intercept storage traffic and possibly make changes

Also consider any VMs that are running management consoles or tools–Should use dedicated management vNICs for

communicationVLANs are often used for network segmentation–Remember that if an attacker gets access to “the

wire” they can sniff all VLANs–Dedicated physical connections may be required for

some specialized security limited functionality (SSLF) environments

Protecting Your Management Communications

Isolating Management – Sharing NICs

Isolating Management – Dedicated NICs

VSHIELD SUITE

vShield is not a single product or featureSuite of several different products–vShield Zones

• Basic VM firewalling–vShield App

• Enhanced VM and grouped firewalling–vShield Edge

• Separation for multi-tenancy environments–vShield Endpoint

• More on this in the next lessonCan be used together for a very granular and flexible virtual security system

An Overview of the vShield Suite

vShield configuration and management handled by a central virtual applianceSelf-contained virtual machine that requires very little maintenanceEasy configuration–Deploy from .ova file–Enter basic configuration from

console via CLI–Rest done from web interface

Using vShield Manager

vShield Zones provides basic stateful firewall functionalityComes with several license levels of vSphere–Advanced, Enterprise, and Enterprise Plus

Allows you to create access lists based on 5-tuple rules– Source Port– Source Address–Destination Port–Destination Address– Protocol

Also can filter Layer 2/3 protocols such as ICMP and ARP

Protecting VMs with vShield Zones

All traffic in and out of a protected VM will pass through a vShield Zone agentEach protected vSphere host will have an agent VM installed and kernel modules loaded–Agent VM uses VMsafe for hypervisor integration

Each protected VM will have a new configuration option added to its .vmx file– ethernet0.filter0.param1 = "uuid=498adeb0-fcfc-db31-b8a7-2fba9881434b.000"

– ethernet0.filter0.name = "vshield-dvfilter-module“

vCenter will add and remove filter lines as needed

How vShield Zones Does Traffic Analysis

Rules can be configured at different levelsEvaluated in a specific order:–Data Center High Precedence Rules–Cluster Level Rules–Data Center Low Precedence Rules–Secure Port Group Rules–Default Rules

New install has a final ANY/ANY rule to allow all traffic

Configuring vShield Zones Firewall Policies

vShield App is not a separate product–Adding a license enhances the functionality of

vShield ZonesSeveral large features over the basic vShield Zones functionality–Group VMs for logical and business reasons–Apply access lists to those logical units– In-depth traffic flow analysis

Traffic flow is the same

Enhancements Provided by vShield App

Used to isolate virtual machines in a port groupUseful to “containerize” a networkAlso provides a number of common services–DHCP–VPN–NAT–Load-balancing

Some use cases include–Multi-tenancy hosting–Site-to-site VPNs amongst

partners or divisions

Using vShield Edge to Provide Multi-Tenancy Security

The vShield suite can seem a bit confusing and overlappingRemember that security is about layersDefense in depth–Perimeter – Use vShield Edge for NAT and VPN access–Application Protection – Use vShield App for application

aware firewalling–VM Protection – Use vShield Endpoint to allow for easy

anti-malware protectionNot all environments will need all pieces–vShield Edge is probably the most optional

Putting Them All Togethers

HELPFUL TOOLS

Many tools available for managing compliance across your environmentCan be simple, can be complexTools from VMware– Host Profiles– Configuration Manager– Two free compliance checking tools (Hardening and PCI)

Third-party tools– Veeam Reporter– HyTrust– LogLogic Compliance Manager– Catbird

Tools for Managing Compliance

From the acquisition of ConfiguresoftCoverage of both physical and virtual systemsCovers all major points of compliance management– Automates deployment of operating systems– Deploy application packages to systems– Monitors systems for configuration drift– Can deploy missing pieces of software, such as AV– Includes many pre-configured templates for things such

as PCI, NIST, HIPAA, Sarbanes-Oxley

vSphere and vCenter integration

About VMware Configuration Manager

VMware offers two free tools– Compliance Checker for vSphere– Compliance Checker for PCI

Available from VMware’s siteCompliance Checker for vSphere compares current configuration against the vSphere Hardening Guidelines– Can check entire cluster at once

Compliance Checker for PCI checks Windows systems against PCI DSS 1.2

Free Compliance Checking Tools

Other ResourcesvSphere Hardening Guide from VMware (as well as other docs):– http://communities.vmware.com/community/vmt

n/server/security?view=documents

Free compliance checker:– http://www.vmware.com/products/datacenter-vir

tualization/vsphere-compliance-checker/overview.html

My Blog: http://www.jasonnash.comTwitter is @nash_JMy Email: jason@varrow.comVarrow Bloggers: http://www.varrowblogs.com

Questions?