managing esxi - tools and techniques

Server Operations and Security TechnologySpeaker: Christopher Janoch

February 1, 2011

ILTA Webinar Session

Migrating to ESXi:Managing without the Console

Tools & Techniques for Managing and Troubleshooting ESXi

Christopher Janoch

Migrating to ESXi:Managing without the Console

Tools & Techniques for Managing and Troubleshooting ESXi

Speaker Bio

Christopher JanochSr. Virtualization Engineer at iVision

Contact Info:[email protected]#cjanoch

iVision, Inc.

VMware’s Big Announcement

VMware vSphere 4.1 and its subsequent update and patch releases are the last releases to include both ESX and ESXi hypervisor architectures.

Future major releases of VMware vSphere will include only the VMware ESXi architecture.

VMware recommends that customers start transitioning to the ESXi architecture when deploying VMware vSphere 4.1

VMware will continue to provide technical support for VMware ESX according to the VMware vSphere support policy

iVision, Inc.

VMware vSphere 4.1 Release Notes – ESX Edition, July 13th, 2010

ESXi Improvements and Changes

Better Reliability and Security- ESXi has fewer security vulnerabilities

Tighter Control over the VMkernal- No arbitrary code or 3rd party agents in core operating system

Lower Development Costs- Only one version of the platform simplifies development and testing

There is no Service Console- Management tools and processes have changed- Learn new systems and procedures *before* migrating systems.

3rd Party Tools must use remote APIs- Backup Agents, Hardware Monitoring Agents ,Reporting Tools

iVision, Inc.

So Why is the Service Console Missing?

iVision, Inc.

VMware ESX Architecture (Version 3.x)

iVision, Inc.

iVision, Inc.

VMware ESX Architecture (Version 4.x)

iVision, Inc.

VMware ESXi Architecture (Version 4.x)

iVision, Inc.

VMware Management Architecture

vSphere SDK

vSphere Web Service API

Common Interface

Module API

Remote CLI

Perl SDK

Java SDK

Power CLI

Agent-lessHardware

Monitoring

VIC Client

Web Browser

Console

VMware vSphere API Management

Common Information Model (CIM)- Agent-less, standards based monitoring of hardware resources

- Output readable by 3rd-Party Management Tools via standard APIs

- VMware and Partner CIM providers for specific hardware devices

vSphere CIM Interface- Enabled detailed hardware health monitoring

- Integrated physical and virtual visibility on server health

- vCenter alarms alert when hardware failures occur

ESXi OEM Models

- VMware, DELL, & IBM versionshttp://www.vmware.com/go/get-free-esxi

- HP Versionhttp://www.hp.com/go/esxidownload

iVision, Inc.

VMware vSphere API Management

Starting with HP SIM 5.3.2 following is available on ESXi:- Network Provider

Ethernet ports information, statistics, port link status and IP and MAC addresses

- Smart Array ProviderController information, storage enclosure and drive cage information, disk drives and spare drives information

- PCI Provider PCI device, adapter card and slot information

- Sensor Provider Temperature Sensors information (for CPU, chassis, Memory), temperature sensors threshold values and current readings

- Software Inventory Ethernet adapter driver versions, CIM provider version and Server Active ROM and redundant ROM versions

iVision, Inc.

So… How do I Manage ESXi?

iVision, Inc.

iVision, Inc.

Management Toolkit

vSphere SDK

vSphere Web Service API

Common Interface

Module API

Remote CLI

Perl SDK

Java SDK

Power CLI

Agent-lessHardware

Monitoring

VIC Client

Web Browser

Console

iVision, Inc.

Management Toolkit

Remote CLI

Perl SDK

Java SDK

Power CLI

VIC Client

Web Browser

Console

Management Toolkit

iVision, Inc.

Virtual Interface Client

VMware vSphere Virtual Interface Client

iVision, Inc.

VMware vSphere Client (VIC)

iVision, Inc.


iVision, Inc.

vCenter Server Login ESXi Server Login


iVision, Inc.

Management Toolkit

iVision, Inc.


ESXi Console

iVision, Inc.

ESXi Console

ESXi Console

Administrative Access- Administrative Password

- Remote Access Lockdown

- Configure Keyboard

Network Management- Automatic Configuration (DHCP)

- Manual Configuration (vmk)

- Restart Networking

iVision, Inc.

ESXi Console

Diagnostics- Test Network Settings

- View System Logs

- Restart Agents

Rescue/Repair- Restart Management Agents

Menu item to reset all management agents including hostd and vpxa

- Reset System ConfigurationMenu item to reset all configuration settingsFix a misconfigured vNetwork Distributed SwitchReset all configurations

- Reset Distributed Switch

- Reboot/Power Off

iVision, Inc.

ESXi Console Configuration

iVision, Inc.

Management Toolkit

iVision, Inc.


ESXi Console

Tech Support ModeESXi Console

ESXi Console: Tech Support Mode

iVision, Inc.

ESXi Console – Tech Support Mode

A command-line interface for diagnosis and repair- Enabled but inactive for default installations- Requires root password to activate- Can be disabled by setting an advanced kernel parameter

http://kb.vmware.com/kb/1003677http://kb.vmware.com/kb/1017910

To be used ONLY for diagnosis and repair- Usually while being guided by VMware Support- Regular usage strongly discouraged

What is “Busybox”?- A single executable containing tiny versions of many common UNIX

utilities- Provides diagnostic tools for any small or embedded system- Busybox is not an operating system, Linux, or a virtual machine

www.busybox.net iVision, Inc.

ESXi Console – Tech Support Mode

iVision, Inc.

Management Toolkit

iVision, Inc.


ESXi Console


ESXi Web Tools

VMware Web-Based ESXi Tools

iVision, Inc.

iVision, Inc.

VMware Web-Based ESXi Tools

Browser-Based Tools

Troubleshooting a Production Environment

- Browser-based Access of Config Fileshttps://esxi01.ivision.lab/host

- Browser-based Access of Log Fileshttps://esxi01.ivision.lab/host/messages

- Browser-based Access of Datastore Fileshttps://esxi01.ivision.lab/folder

iVision, Inc.

Management Toolkit

iVision, Inc.


ESXi Console

Remote CLI


ESXi Web Tools

VMware vSphere Remote Command Line

iVision, Inc.

vSphere Remote Command Line (rCLI / vCLI)

iVision, Inc.


rCLI / vCLI Commands

- Host ConfigurationNTP, SNMP. Remote SYSLOG, ESX Configuration, Kernel Modules, Local Users and Security

- Storage ConfigurationNAS, SAN, iSCSI, vmkfstools, Storage Pathing, VMFS Volume Management

- Network ConfigurationvSwitches (Standard & Distributed), Physical NICs, Vmkernal NICs, DNS Settings, Routing Tables

- MiscellaneousMonitoring, File Management, VM Management, Host Backup/Restore & Update

CLI Community sitehttp://communities.vmware.com/community/developer/vsphere_cli

iVision, Inc.


rCLI/vCLI Commands can be Executed Directly on the ESXi Host

- vCLI uses API to connect directly to hostd- Uses local users and roles defined on each individual ESX Server- Commands do not get logged in vCenter- Some commands only work this way:

vicfg-snmp, vifs, vicfg-user, vicfg-cfgbackup, vihostupdate, vmkfstools, esxcli

rCLI/vCLI Commands can also be Executed through vCenter

- vCLI uses vSphere API to connect to vCenter- Uses global users and roles defined for vCenter- Commands get logged in vCenter events database

Only commands which actually change state, not commands which simply query state

iVision, Inc.

Management Toolkit

iVision, Inc.


ESXi Console

Perl SDK Toolkit

Remote CLI


ESXi Web Tools

VMware Perl SDK

iVision, Inc.

VMware vSphere Management Assistant

What is vMA?- A free CentOS (Linux) based Virtual Appliance that you can

download from VMware and import into your infrastructure.http://www.vmware.com/support/developer/vima/http://www.vmware.com/support/developer/vima/vma41/doc/vma_41_guide.pdfhttp://communities.vmware.com/docs/DOC-10878

- Allows administrators and developers to run scripts and agents to manage ESX/ESXi 3.5 Update 2 or later, ESX/ESXi 4.0 and 4.1 and vCenter Server 4.0 and 4.1 systems.

- There is no GUI interface – vMA is a command-line tool designed to act as a Service Console

- Contains pre-packaged software includingvi-fastpass (VMware Authentication component) vi-logger (VMware Logging Component)

iVision, Inc.


vMA 4.1 Software includes:

- CentOS release 5.3 64-bit Enterprise LinuxProvides Linux command set functionality

- vSphere CLI 4.1Commands for managing vSphere from the command line

- vSphere SDK for Perl 4.1Client‐side Perl framework that provides a scripting interface to the vSphere API, including utility applications and samples for common tasks.

- SMI‐SvMA includes the VMware implementation of the CIM profiles compatible with the Storage Management Initiative Specification (SMI‐S 1.0.2)

- Java JRE version 1.6Runtime engine for Java‐based applications built with the vSphere Web Services SDK.

- VMware Tools (current version)Provides an interface to the hypervisor operating system

iVision, Inc.

iVision, Inc.


Hardware/Software Requirements:- ESX/ESXi Host that supports 64-bit virtual machines- Intel Processors with EM64T support with VT enabled- 512MB available memory- vSphere Client 4.0 or later for deployment

Installing vMA1. Use a vSphere Client to connect to a system that is running ESX/ESXi 4.1, ESX/ESXi 4.0, ESX/ESXi 3.5

Update 2 or later, or vCenter Server 4.0.

2. Select File > Deploy OVF Template. Browse, select the OVF, and click Next.

3. Click Next when the download details are displayed, and accept the license agreement.

4. Specify a name for the virtual machine, and a location when prompted.

5. If connected to a vCenter system, select the resource pool for the virtual machine. By default, the top‐level root resource pool is selected.

6. If prompted, select the datastore to store the virtual machine on and click Next.

7. Select the network mapping (the management network on which vCenter and ESX/ESXi systems to be managed are located) and click Next.

8. Review the information and click Finish. The wizard deploys the vMA virtual machine.

9. Log in to vMA the first time an configure Network settings and user Passwords.


iVision, Inc.


vSphere SDK for Perl comes bundled with rCLI / vCLI

Contains pre-packaged utilities for ESXi Management:

- hostdiagnostics.plExtracts the specified log from the host of the virtual center

- hostops.pladd, remove, disconnect, reconnect, enter/exit maintenance mode, reboot

- sharesmanager.plDisplays or modifies shares for memory, cpu, and disk for specified VMs

- snapshotmanager.plCaptures the state of one or more virtual machines into a snapshot

- viperformance.plRetrieves performance counters from a host.

Perl Utility Application Referencehttp://www.vmware.com/support/developer/viperltoolkit/viperl40/doc/vsperl_util_index.html

iVision, Inc.

iVision, Inc.

vMA Command Set

Management Toolkit

iVision, Inc.


ESXi Console

Perl SDK Toolkit

PowerCLI

Remote CLI


ESXi Web Tools

VMware PowerCLI (PowerShell)

iVision, Inc.

VMware vSphere PowerCLI

A command-line and scripting interface that leverages the Windows PowerShell infrastructure

- Object-oriented functions allows you to manipulate and examine objects instead of parsing strings, values, and integers

- Cmdlets are simple, intuitive and can be piped into a long chain to accomplish complex management tasks easily

- Large, extremely active user community that is enhancing our base implementation with new scripts that are shared on the forum

- As VMware conceives of new features and services in vSphere, we will simultaneously design a PowerShell interface

- http://vmware.com/go/powercli

iVision, Inc.

Sample Perl Script: List VMs

useVMware::VILib;Opts::parse();Opts::validate();Util::connect();my$datacenter=“.”my$datacenter_view=Vim::find_entity_view(view_type=>’Datacenter’,

filer=> { name=>$datacenter });my$host_views=Vim::find_entity_views(view_type=>’HostSystem’,

begin_entity=>$datacenter_view);my$counter=1;print”Hosts found/n”;foreach (@host_views) {

print”$counter: “ . $_->name . “\n”;$counter++;

}$counter=1;print”\nVM’s found:\n”;my$vm_views=Vim::find_entity_views(view_type=>’VirtualMachine’,

begin_entity=>$datacenter_view);foreach (@$vm_views) {

print”$counter: “ . $_->name . “\n”;$counter++;

}Util::disconnect();

iVision, Inc.

Sample PowerCLI Script: List VMs

Get-VIserver myserverGet-Vm

iVision, Inc.

iVision, Inc.

PowerShell Requirements

Hardware/Software Requirements:- Windows XP, Vista, Windows 7, Server 2003, Server 2008 or later- .NET Framework 2.0 or later installed (and patched!)

On recent Operating Systems, PowerShell is already Installed- 32-bit Systems: Windows\systems32\WindowsPowerShell\v1.0- 64-bit Systems: Windows\SysWOW64\WindowsPowerShell\v1.0

PowerShell can be Downloaded from Microsoft- http://www.Microsoft.com/Powershell

- Be sure to download the correct build for your version of Windows

VI Toolkit

What is the VI Toolkit?- The Virtual Infrastructure Toolkit for Windows comes with over 120

cmdlets for Windows PowerShell- VMware-PowerCLI-4.1.1-332441 Current Version

VI Administration, Deployment Tasks, Maintenance Routines, etchttp://www.vmware.com/developer/http://www.vmware.com/download/sdk/http://vmware.com/go/powershell

Hardware/Software Requirements:- 5 MB free space- VMware Virtual Infrastructure ESX 3.x or later- PowerShell versions 1.0 or 2.0

iVision, Inc.

VMware vSphere PowerCLI

iVision, Inc.

PowerCLI Command Set

iVision, Inc.

Management Toolkit

iVision, Inc.


ESXi Console

Perl SDK Toolkit

PowerCLI

Remote CLI

3rd-Party Tools


ESXi Web Tools

3rd-Party Tools and Plug-ins

iVision, Inc.

3rd-Party Tools

Monitoring Tools

- VKernel CapacityVIEWhttp://www.vkernel.com/products/capacity-viewCapacityVIEW collects CPU, memory and storage allocations and utilization levels. It then analyzes this data to provide you with desktop Capacity alerts

- Quest vFoglight QuickViewhttp://vizioncore.com/free/vFoglight-quick-view/ vFoglight QuickView provides essential alerts and data to help you rapidly detect, diagnose, and resolve critical issues within the virtual infrastructure

- Veam Reporter Free Editionhttp://www.veeam.com/vmware-esx-reporter-free.htmlVeeam reporter manages enterprise reporting, change management and capacity planning

iVision, Inc.

3rd-Party Tools

Configuration Tools

- RVTools 3.0http://robware.net/A Windows .NET 2.0 application which uses the VI SDK to display information about your virtual machines and ESX hosts

- Aligned Vdisk Finder

- Vdisk Waste Finderhttp://read.virtualizeplanet.com/Windows Application to examine VM virtual disk files to determine alignment and capacity usage

iVision, Inc.

3rd-Party Tools

Communication Tools

- Veeam FastSCPhttp://www.veeam.com/vmware-esxi-fastscp.html Veeam FastSCP can copy files over 6 times faster than ther SCP-based tools as it uses full network capacity. It also features traffic compression and empty block removal for best file copy performance.

- WinSCPhttp://winscp.net/eng/index.php

- VMX Trilead VM Explorerhttp://www.trilead.com/Trilead VM Explorer is a management tool that eases management, backup and disaster recovery tasks in your VMware ESX environment.

iVision, Inc.

So How Do I Choose?

iVision, Inc.

What Tools To Use?

vSphere rCLI- Can configure “everything” about the ESXi host- Can download, modify, and upload configuration files

vSphere Perl SDK- Can script vSphere inventory configurations

Join to vCenter, Join Cluster, etc.- Can run script flies

vSphere PowerCLI- Can perform wide range of configuration tasks

iVision, Inc.

What Tools To Use?

Case #1: - Users that are completely brand new to managing vSphere

- Comfortable with the Windows OS/GUI environment

- Administrators looking for quick and immediate solutions

vSphere Client, 3rd-Party Tools and Tech Support Mode- These tools are easier to “figure out” the commands as needed

- The vSphere Client provides a familiar GUI environment

- Use PowerCLI for more advanced control and Automation

PowerShell can be used to manage VMware and Microsoft virtualization solutions

iVision, Inc.

What Tools To Use?

Case #2: - Users that have worked on the Service Console since the early days

of ESX 2.x, 3.x and 4.0

- Users who are very familiar with the esxcfg-* commands

- Users who have a strong UNIX/Linux background

vMA with vCLI and Perl SDK tool sets- If you’re more comfortable with UNIX/Linux, then using vMA may

make more sense

- By leveraging these familiar commands and familiar output, it is much easier to migrate any scripts you may have already created

- vMA are still needed for things like resxtop and logging collection

- Large libraries of Perl scripts are available online for use and modification

iVision, Inc.

What Tools To Use?

Case #3:

- Enterprise Developers and Application Writers

vSphere SDK for .NET, Java, and Web Services

- Perl and PowerCLI were primarily developed as Scripting environments

- VMware also supplies SDK libraries for Java and .NET which are available for download

http://communities.vmware.com/community/developer/downloads

http://www.vmware.com/support/developer/vc-sdk/

http://communities.vmware.com/community/developer/forums/java_toolkit

iVision, Inc.

Questions?

iVision, Inc.

Example Procedures

- NTP Time SynchronizationvSphere Client

– Configuration Tab Time Configuration Properties

- Configure Management Network SettingsDirect User Console

– F2 Configure Management Network

- Configure vNIC ParametersvSphere Client

– Configuration Security Properties Local Tech Support Options Start

User Console – F2 Troubleshooting Options Enable Local Tech Support

User Console – Tech Support Mode– Alt-F1 Login esxcfg-nics vmnic0 –s 1000 –d full

- Review Log FilesBrowser-Based Access

– https://1.100.24.101/host/messages

iVision, Inc.

Example Procedures

- Backup & Restore Configuration of an ESXi hostvCLI Command

– perl vicfg-cfgbackup.pl –server HOSTNAME –username root –s c:\location\server.txt– perl vicfg-cfgbackup.pl –server HOSTNAME –username root –l c:\location\server.txt

- ESXi Performance TroubleshootingvMA

– resxtop –server HOSTNAME –username root

- Syslog: Centralized log collectionvMA

– Sudo vifp addserver HOSTNAME– Sudo vifp listservers– Vilogger enable --server HOSTNAME

WinSCP– /var/log/vmware/

- List VMs with current SnapshotsPowerCLI

– Connect-VIserver vCENTERHOSTNAME– get-vm \ get-snapshot \ format-table name,description -autosize

iVision, Inc.

Upcoming ILTA Webinar Sessions

Look Ahead for More Information:

Migrating to ESXi – How ToStrategies, Procedures, and Precautions

Encore Session:Late February

Contact Info:[email protected]#cjanoch

managing esxi - tools and techniques

Documents

security esxi

esxi improvements

standard apis vmware

esxi hypervisor architectures

deployingvmware vsphere

party management tools

hardware monitoring

party agents