it management suite design, tuning & optimization do it

ITMS Performance Tuning & Optimization 1

IT Management Suite Design, Tuning & Optimization – Do it Right the First Time

Paul Conover, Brian Sheedy Dir, SQA Engineer, Sr Field Enablement Manager

SYMANTEC VISION 2013

Agenda


Thinking About Performance 1

Architecture 2

SQL Server 3

Infrastructure & Product Considerations 4

Operating System & IIS 5

SYMANTEC VISION 2013 ITMS Performance Tuning & Optimization 3

Thinking About Performance “Any information regarding pre-release Symantec offerings, future updates or other planned modifications is subject to ongoing evaluation by Symantec and therefore subject to change. This information is provided without warranty of any kind, express or implied. Customers who purchase Symantec offerings should make their purchase decision based upon features that are currently available.”

Expectations: this is a white paper disguised as a presentation


ITMS Performance Tuning & Optimization Thinking About Performance

• Random server errors

• Server timeouts

• Query timeouts

• Slow operations

• Queues filling up

• Caching warnings in the log

• Excessive deadlock retries in the log

• Key Performance Counters (Disk, Wait Stats)


Obvious Signs for Tuning


Architecture


ITMS Performance Tuning & Optimization Architecture

• Engage the Right People

– Physical Server & OS Ops

– Virtual Server Ops

– Network Ops

– Storage Ops

– Database Administrators

– Users

– Business Requirement Owners


Successful Implementation • Common Architecture Failings

– Incorrectly matched architecture for implementation

– Insufficient configuration for architecture

– Rollout didn’t match design

– Excessive non-essential data movement due to incorrect use case analysis

– Site configuration deficiencies

– IO & resource deficiencies

– Ongoing alignment with use cases and business requirements

• Growth

• Security, compliance & regulation



•Recommended Architectures in the Guide – ITMS Server (20 k per server, 120 k per hierarchy, 6 children)

– Site Server (7.5k nodes per Task/Pkg, 5k/PXE, 600 sites per ITMS-Task/pkg/PXE)

– 3000 Sites per hierarchy


Planning Server Implementation with the Guide

• Meet System Requirements

• Same Data Center Preferred

• Plan Sites

• Plan CEM



• Plan, Design, Configure, and Monitor Sites & Site Health

• Favor Sites, and Org Groups with Security Over Hierarchy

• Package Servers

– Constrain Packages Servers

– Cascade Package Staging – Unconstrained -> Constrained -> Agents

• Task Server

– Limit Task Server Communications on the ITMS Server

– Provides Connection Management & Request Consolidation

• not bandwidth

– Don’t have agents connect to the parent

• Generally - Have at least 1 task server


Site Design Planning



• Up to 3,000 concurrent connections, 60000 CEM-enabled endpoints

• Hardware requirements:

– Preferably physical box, 8GB RAM, 40GB HDD and dual-core CPU

– VM-based IG – Remember VM Latency/Resources


ITMS 7.5 CEM Design (See the CEM Sessions for Deep Dive)


ITMS Performance Tuning & Optimization Architecture - Virtual Machines

• Resource Availability to ITMS

• CPU, RAM, Disk, Network

• VM Resource Latency

• Virtual latency will be manifest to users as application latency

• Resource storms

• Vmotion, SnapShots & Defrag/MBR

• Example: Host with 4 CPUs

• ITMS Server configured for 4 CPUs

• Second App server configured for 1 or more CPUs

• Both will have have dramatic and measureable latency on CPU

• Well Balanced VM Infrastructure may still see ~20 less efficiency than physical


Virtual Environment Considerations


SQL Server


ITMS Performance Tuning & Optimization SQL Server

• Right-size SQL Resources

– Exceed IOPs

– Exceed Memory

– Meet CPU


Storage IO & Resources

Component POC 100 – 1,000 1,000 – 5,000 5,000 – 10,000 10,000 – 20,000

Processors 1 core 4 cores 8 cores 8-16 cores 16+ cores

Disk Speed (in IOPS)

180 - C: OS,SMP 200 - D: SQL

180 - C: OS,SMP 200 - D: SQL

180 - C: OS, SQL App 300 - D: SQL DB 300 - E: Logs 200 - F: TempDB



Disk Capacity 80 GB 80 GB 120GB 250 GB 400 GB

RAM 16 GB 16 GB 16+ GB 24+ GB 32+ GB

• Estimate 6-8MB DB storage per managed endpoint

– Implement a Database that uses the above calculation + 10%

– i.e. 5,000 Endpoints x 6MB = 30GB + 3GB = 33GB CMDB minimum



• Design for High IOPs – Performance Device

• Storage Selection (SSE/PCIe, SAN, Hybrid, Spindle)

– SSD/SAN: > 1 million IOPS

• High IO Bandwidth

• Low IO Latency

– Single Spindles: 150-200 IOPs

• RAID and multiple channels

– Optimize capacity requirements for Speed

• Say NO to BIG, SLOW storage from SAN admins.

• SSD or PCIe such as FusionIO DUO

– Test the storage subsystem with IOMeter and/or SQLIO


Storage Subsystem



• http://www.wesworld.net/raidcalculator.html

• If spindles: fast array of many small disks

• RAID 1+0, RSID 5 - Temp DB Data, CMDB Data, CMDB Logs 1+0, Temp Logs RAID 0

Storage RAID Calculator

Presentation Identifier Goes Here 14

http://www.wesworld.net/raidcalculator.html






Test the Storage Subsystem - IOMeter/SQLIO

3 Bias Patterns

Large Test Data

64/128/256K Alignment

8/64k Transfer

8/64k, Reply

8/64k Request Size

100% Random Distribution

Burst Length = 1



• Plan Volume Creation & Data Separation

– Worth the attention -- especially with spindle drives

• SQL Best Practices: http://technet.microsoft.com/en-us/library/cc966534.aspx

• Optimize for I/O parallelism

– Stripe size: 64/128 KB on SQL 2005 or 64/128/256 KB for SQL 2008+

– Spindles: create multiple files, place on separate physical volumes • CMDB Data

• CMDB Logs

• TempDB Data

• TempDB Logs

• High IO SSD: can place files on same volume.

– Again, Test the storage subsystem with IOMeter and/or SQLIO


File Management

http://technet.microsoft.com/en-us/library/cc966534.aspx







• Multiple Database Files – Myth vs Reality

– http://blogs.msdn.com/b/psssql/archive/2007/02/21/sql-server-urban-legends-discussed.aspx

– Not for IO thread per CPU.

– For Latency + I/O

• Case for Multiple Data Files

– http://www.sqlskills.com/blogs/paul/benchmarking-multiple-data-files-on-ssds-plus-the-latest-fusion-io-driver

– Single Spindle Drive – less value due to total disk throughput.

– SSD – Sweet spot: # files = # CPUs < 8

– Too many can degrade

• Equal File Size to Avoid Ratio Dist.


File Management - Case for Multiple Data Files

http://blogs.msdn.com/b/psssql/archive/2007/02/21/sql-server-urban-legends-discussed.aspx










http://www.sqlskills.com/blogs/paul/benchmarking-multiple-data-files-on-ssds-plus-the-latest-fusion-io-driver



























• Contention on I/O, syscolumns, sysobjects

• As many files as schedulers

– At least two

– Large Scale Systems, consider # schedulers = # CPUs < 8

• Enable Uniform Allocations: Trace Flag (-T1118) – Avoids contention on SGAM (even on SQL 2005+)

• Monitor SQL IO stats

– PAGELATCH_XX

– PAGEIOLATCH_XX

• Optimize TempDB for Server Restarts (Growth & Size) ITMS Performance Tuning & Optimization 18

File Management–TempDB


SQL Helper File – The “Goods”

• Do NOT run all queries at once.

• Do choose to run queries one by one if you desire to use the functionality it provides.

• Do test.

Attached file has queries to review settings & set configuration, and Measure Perf




• Use Performance Monitor and/or SQL Server Wait Stats

– Latency: < 20 ms (See query)

– Measure IOPs: http://sqlserverio.com/2011/02/08/gather-virtual-file-statistics-using-t-sql-tsql2sday-15/

• DiskQLength > 1 or 2

– Can be an indicator

– Can be misleading due to SQL asynchronous IO.

– Watch Average Disk Sec/Read Write & Transfer

• Memory

– Page Life Expectancy: > 300*

– Buffer Cache Hit Ratio: > 95-98%

– Free Pages: > 640


Monitoring IO – How to Check IOPs & Latency

http://sqlserverio.com/2011/02/08/gather-virtual-file-statistics-using-t-sql-tsql2sday-15/




















• ITMS large and complex queries can over-utilize CPUs due to parallelism

– http://blogs.msdn.com/b/jimmymay/archive/2008/11/28/case-study-part-1-cxpacket-wait-stats-max-degree-of-parallelism-option-introduction-to-using-wait-stats-to-identify-remediate-query-parallelism-bottlenecks.aspx

• Default = 0 (change it) – Uses all CPUs

– 2 to 4 on Parent (1x6)

– 1 to 2 on Child

– Never > 8 or # CPUs, whichever is less.

• Use CXPACKET Wait Stats to verify SQL parallelism

– Use wait type query on the next slide

– Or, SSE Tools: http://symantec.com/docs/howto60787

– Consider changing if CXPACKET > 5%

• Increase Cost Threshold for Parallelism to 10

– Reduces chance of simple plans going parallel ITMS Performance Tuning & Optimization 21

Max Degree of Parallelism

http://blogs.msdn.com/b/jimmymay/archive/2008/11/28/case-study-part-1-cxpacket-wait-stats-max-degree-of-parallelism-option-introduction-to-using-wait-stats-to-identify-remediate-query-parallelism-bottlenecks.aspx













































http://symantec.com/docs/howto60787




• On x32, use AWE. Don’t use AWE on x64 SQL

– x32 SQL with AWE, /3Gb, /pae, etc

– AWE is deprecated for x64: http://msdn.microsoft.com/en-us/library/ms175581(v=sql.105).aspx

• Enable the option to Optimize for Ad hoc Workloads

– ITMS has high degree of Ad-hoc/dynamic queries that are non-cacheable

– Doesn’t waste caching on Single-Use queries

– Caches on second+ request

• Set Maximum Server Memory value

– In 'Server memory options' page

– leave enough memory for the Operating System and tools such as SQL Management Studio to run effectively

– e.g. around 4-6 GB less than the amount of RAM installed.


SQL Memory Configurations

NSQLRam = NTotalRam – (NOSRam + NAppRam)

http://msdn.microsoft.com/en-us/library/ms175581(v=sql.105).aspx






• Enable Arithmetic Abort in the Server/Instance Properties

– Terminates a query when a overflow or divide-by-zero error occurs.

– http://msdn.microsoft.com/en-us/library/ms190306.aspx

• Enable Read Committed Snapshot mode for improved UI *** • Improve reporting& console performance without blocking other activity

alter database Symantec_CMDB set read_committed_snapshot on with rollback after 30;

• Disable Autoshrink and Autogrow, Control Growth Settings 500 MB

– (default for ITMS Creation).

– Avoids unnecessary file re-allocations latency and increases file fragmentation.

– Avoids unnecessary CPU and IO cycles.


Other SQL Settings

http://msdn.microsoft.com/en-us/library/ms190306.aspx






• Update Statistics (nightly)

– Ensures query processor makes “the best” choices it can given ITMS implementation.

– Or, enable Auto Update Statistics Asynchronously • Memory Leak in SQL - get latest update

• Database Properties > Options page.

• Enabling affects your ability to put a database into single-user mode (For Maintenance)

• Regular Database Page Maintenance

– Rebuild Indexes (Monthly)

– Reorganize pages (Weekly)

• Regular backup

– Simple Recovery Mode • Yields Lower IO but you should have an aggressive backup strategy

– Vs. FULL logging with backups


Maintenance Plan


Infrastructure & Product Considerations


ITMS Performance Tuning & Optimization Infrastructure & Product Considerations

• Frequency & Node Counts

• Tasks & Policies

– Reduce targeting & scheduling, frequency and results

– All types: Custom Inventory, Software, Inventory, Patching, …

• Reports

– Optimize your reports that you run often

– Query lowest level resource views in the resource type hierarchy

• Filters

– Optimize all filters you create that get used

– Query lowest level resource views in the resource type hierarchy

• Inventory Solution

– Limit frequency & Amounts to only what is needed.


Customizations & Usage




Product Configuration

Setting < 1000

endpoints 1000 – 5000

5000 – 10,000

10,000 – 15,000

15,000 to 20,000

Agent Configuration Interval Every 1 hour Every 1

hour Every 2 hours

Every 3 hours

Every 4 hours

Full inventory collection schedule Monthly Monthly Monthly Monthly Monthly

Delta inventory collection schedule Weekly Weekly Weekly Weekly Weekly

Complete Resource Membership Update

Daily Daily Daily Daily Daily

Delta Resource Membership schedule

15 min 20 min 30 min 45 min 1 hr

Task Service task update interval 5 min 5 min 15 min 15 min 15 min

Policy Refresh schedule 5 min 10 min 15 min 15-20 min 15-20 min



• Cache size warning in the log

– Source: Altiris.NS.Security.SecurityCache.HandlePermissionCacheFloodedDescription: Core setting SecurityCacheSize is insufficient to handle incoming security checks. Current Size:xxx

• Bump up cache sizes

– CodeBaseCacheSize: NumPkgs * NumPkgServers

– ItemClassCacheSize: 1000 or Number of rows in Class table

– ItemCacheSize: 3000 to 100,000

– SecurityCacheSize: 5000 to 100,000(800,000 on full parent)

– StringCacheSize: 1,000,000 or # Rows in String table

– ResxCacheSize: # Strings in ResourceStrings table

– ClientPolicyCacheMaxSize: NumPolicies + Growth of Policies


Core Settings – Cache Sizes



• ResourceAssociationTablelockThreshold: 1,000,000

• ReportMaxRowCount from: 50,000 to 500,000

– Generally, set it to 0 unless performance is an issue

– Base on the use case.

• Replication Settings - Latest 7.1 SP2 MP1 update v2+

– ReplicationDataBatchSize: 5,000 – 10,000

– ReplicationEventBatchSize: 5,000 – 10,000

– ItemStateMaxSizeInKB: 128MB

• size of the data container to allow for the increase in the Batch Sizes


Other Core Settings



• SuppressReportAutorun (ITMS 7.5)

– When enabled reports do not run immediately.

– Affects every report.

• When enabled you must click Refresh to get results.

• Disabled by default


Other Core Settings - Report Auto Run Control



• Use Saved Searches for frequently used search

• Turn off Search as you type

• Limit result lists

– < 20,000 (consider < 500)


Enhanced Console View Settings



• Tune it, don’t accept out-of-the-box.

– Don’t schedule full replication (includes re-replication of full security)

• Do FULL rarely/manually (mostly in troubleshooting)

– Leverage Differential Replication Schedules (first differential is a full)

• KB that shows how to do full security in Delta

– Always stagger replications: Child 1 @ 1 am, Child 2 @ 3 am

• Leave time for success (avoid schedule clashes)

• Also stagger with other heavy activities such as full inventory.

• Limit what gets replicated

– Event replication – Only data that you must act on.

– Leverage IT Analytics over replication.

– Avoid extraneous replicated resources & data -- ie Software Files


Hierarchy



• Change log locations

• Move the event queues

• Don’t compress Event Queue and Replication path

• Use File System junction points to move Replication data folder

• Tune SMP logging & Managed Delivery verbose logging

– Use as needed

• Clear profile trace after each capture, don’t leave it running

• Use history judiciously

– Be cautious what data you keep and the duration that you keep it.

– Consider adding indexes if your history is impacting operations (such as viewing the resource or resource deletion).


Operating Environment


Operating System & IIS


ITMS Performance Tuning & Optimization Operating System & IIS

• Optimize Performance for Background Services

• Control OS Background Activities

– Restrict schedules

– Limit directories and file types: *.MDF,*.LDF,*.TMP,*.NSE, Replication & Log Folders

– Services to Consider

• Indexing Service

• Encryption

• Compression

• AV Scanning


Operating System



• Stop unnecessary services.


Operating System



• General .Net Assembly functionality

– Published .Net Assemblies DLLs are digitally signed

– Signatures are validated when assemblies are loaded

• Certificate Revocation List validation at crl.microsoft.com

– 30-60 second timeout delay for each assembly load

• ITMS Symptoms

– Windows services on the Platform computer sometimes fail during startup.

– Some Symantec Management Console pages take a very long time to load.

– The SMP may experience performance issues after being offline for 15 days

– Applies to Site Servers also

• Resolution

– Disable CRL lookups

– You can re-enable CRL lookups easily

– Apply to Remote Site Servers

– Via Hosts File, Registry, or Machine.config


.Net Framework CRL Validation –Server or Site has no Internet

Edit the x86 & x64 Machine.config Files:

<configuration>

<runtime>

<generatePublisherEvidence enabled="false"/>

</runtime>

</configuration>



• IIS data compression

– Not especially effective since polices are already compressed but, can give some improvements for other activities

• Issues with Kerberos vs NTLM see KB on Browser settings for IE

– http://support.microsoft.com/kb/2749007

– Disable IE Pre-Authentication Headers on Machines accessing the console • OR

– Turn Off Kerberos Authentication in IIS

• HTTP Logging (disable or reduce)


IIS & Internet Explorer

http://support.microsoft.com/kb/2749007

http://support.microsoft.com/kb/2749007


Topic Links

Planning & Implementation Guide

• http://www.symantec.com/business/support/index?page=content&id=DOC4827

SQL IO Tuning

• http://sqlserverio.com/2011/02/08/gather-virtual-file-statistics-using-t-sql-tsql2sday-15/

• http://www.sqlskills.com/blogs/paul/how-to-examine-io-subsystem-latencies-from-within-sql-server/

• http://blogs.msdn.com/b/askjay/archive/2011/07/08/troubleshooting-slow-disk-i-o-in-sql-server.aspx

IOMeter/SQLIO Tools

• http://www.microsoft.com/en-us/download/details.aspx?id=20163

SSE Tools • http://symantec.com/docs/howto60787

References & Resources


http://www.symantec.com/business/support/index?page=content&id=DOC4827

http://www.symantec.com/business/support/index?page=content&id=DOC4827


















http://www.sqlskills.com/blogs/paul/how-to-examine-io-subsystem-latencies-from-within-sql-server/



















http://blogs.msdn.com/b/askjay/archive/2011/07/08/troubleshooting-slow-disk-i-o-in-sql-server.aspx















http://www.microsoft.com/en-us/download/details.aspx?id=20163




Topic Links

Task Server Practices

• http://www.symantec.com/business/support/index?page=content&id=HOWTO48411

Package Staging Practices

• http://www.symantec.com/connect/articles/how-handle-getpackageinfo-request-storms

• http://www.symantec.com/connect/blogs/working-codebasecache-post-getpackageinfo-series

Constrained Pkg Servers


• http://www.symantec.com/connect/forums/constrained-vs-unconstrained

Package Replication

• http://symantec.com/docs/howto60787



References & Resources


http://www.symantec.com/business/support/index?page=content&id=HOWTO48411


http://www.symantec.com/connect/articles/how-handle-getpackageinfo-request-storms









http://www.symantec.com/connect/blogs/working-codebasecache-post-getpackageinfo-series











http://www.symantec.com/connect/forums/constrained-vs-unconstrained










Thank you!

Copyright © 2012 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.


Paul Conover

[email protected]

801-228-7119


How To Print Slides Without a Dark Background

• Conserve printer ink by printing in Grayscale mode. Black backgrounds will be inverted to white, white text to black

• To View how your slides will print in Grayscale mode:

– From the View tab, in the Color/Grayscale group, click Grayscale

– Your Slides will display without color. Check each slide to ensure that all objects are displaying at high contrast (black or gray on white background)

– You can make adjustments for printing purposes. (Note: making changes in Grayscale does not effect the color on your slides)

– Select Back to Color View when finished


• To adjust Grayscale settings:

– The Grayscale tab includes various settings: Automatic, Grayscale, Light Grayscale, Inverse Grayscale, Gray with White Fill, etc.

– To apply a different grayscale setting, select the object(s) and choose a setting in the Change Selected Object group on the Grayscale tab

• To Print slides in Grayscale mode:

– From the Microsoft Office Button (top left corner), click Print to access the Print dialog box

– Under Color/grayscale: select Grayscale

it management suite design, tuning & optimization do it

Documents