Brian Sheedy Sr. Principal Technical Education Consultant – Endpoint Management

ITMS 7.5 Performance Tuning & Optimization

1

DISCLAIMER: This documentation is provided "as is“ and all express or implied conditions, representations and warranties, including any implied warranty of merchantability, fitness for a particular purpose or non-infringement, are disclaimed, except to the extent that such disclaimers are held to be legally invalid. Symantec corporation shall not be liable for incidental or consequential damages in connection with the furnishing, performance, or use of this documentation. Always evaluate changes to your implementation in a test environment before applying them to a production installation

2

Agenda

Designing for Performance 1

SQL Server 2

Infrastructure & Product Considerations 3

Operating System & IIS 4

3

Designing for Performance

4

Obvious Signs for Performance Tuning

• Random server errors

• Server timeouts

• Query timeouts

• Slow operations

• Queues filling up

• Caching warnings in the log

• Excessive deadlock retries in the log

• Key Performance Counters (Disk, Wait Stats)

5

When Implementing ITMS 7.5…

• Engage the Right People

– Physical Server & OS Ops

– Virtual Server Ops

– Network Ops

– Storage Ops

– Database Administrators

– Users

– Business Requirement Owners

• Common Architecture Failings

– Incorrectly matched architecture for implementation

– Insufficient configuration for architecture

– Rollout didn’t match design

– Excessive non-essential data movement due to incorrect use case analysis

– Site configuration deficiencies

– IO & resource deficiencies

– Ongoing alignment with use cases and business requirements

• Growth

• Security, compliance & regulation

6

Planning your ITMS 7.5 Implementation

•Use Recommended Architectures in the Guides – ITMS Server (20 k per server, 120 k per hierarchy, 6 children)

– Site Server (5k Task/Pkg, 600 sites per NS)

– 3000 Sites per hierarchy

• Meet Requirements

• Same Data Center

• Plan Sites

• Plan CEM

7

Site Server Design

• Plan, Design, Configure, and Monitor Sites & Site Health

• Favor Sites, and Org Groups with Security over Hierarchy

• Site Servers w/Package Servers

– Constrain Package Servers

– Cascade Package Staging – Unconstrained -> Constrained -> Agents

• Task Server

– Limit Task Server Communications on the ITMS Server

– Provides Connection Management & Request Consolidation

• not bandwidth

– Don’t have agents connect to the parent

• Generally - Have at least 1 task server

8

ITMS 7.5 in Virtual Environments

• Resource Availability to ITMS

• CPU, RAM, Disk, Network

• VM Resource Latency

• Virtual latency will be manifest to users as application latency

• Resource storms

• Vmotion, SnapShots & Defrag/MBR

• Example: Host with 4 CPUs

• ITMS Server configured for 4 CPUs

• Second App server configured for 1 or more CPUs

• Both will have have dramatic and measureable latency on CPU

• Well Balanced VM Infrastructure may still see ~20 less efficiency than physical

9

SQL Server Tuning

10

Storage, IO & Resources

• Right-size SQL Resources

– Exceed IOPs

– Exceed Memory

– Meet CPU

Component POC 100 – 1,000 1,000 – 5,000 5,000 – 10,000 10,000 – 20,000

Processors 1 core 4 cores 8 cores 8-16 cores 16+ cores

Disk Speed (in IOPS)

180 - C: OS,SMP 200 - D: SQL

180 - C: OS,SMP 200 - D: SQL

180 - C: OS, SQL App 300 - D: SQL DB 300 - E: Logs 200 - F: TempDB

180 - C: OS, SQL App 400 - D: SQL DB 400 - E: Logs 300 - F: TempDB

180 - C: OS, SQL App 600 - D: SQL DB 600 - E: Logs 400 - F: TempDB

Disk Capacity 80 GB 80 GB 120GB 250 GB 400 GB

RAM 16 GB 16 GB 16+ GB 24+ GB 32+ GB

• Estimate 6-8MB DB storage per managed endpoint

– Implement a Database that uses the above calculation + 10%

– i.e. 5,000 Endpoints x 6MB = 30GB + 3GB = 33GB CMDB minimum

11

Storage Subsystem

• Design for High IOPs – Performance Device

• Storage Selection (SSD/PCIe, SAN, Hybrid, Spindle)

– SSD/SAN: > 1 million IOPS

• High IO Bandwidth

• Low IO Latency

– Single Spindles: 150-200 IOPs

• RAID and multiple channels

– Optimize capacity requirements for Speed

• Say NO to BIG, SLOW storage from SAN admins.

• SSD or PCIe such as FusionIO DUO

– Test the storage subsystem with IOMeter and/or SQLIO

12

Test the Storage Subsystem - IOMeter/SQLIO

• 3 Bias Patterns

• Large Test Data

• 64/128/256K Alignment

• 8/64k Transfer

• 8/64k, Reply

• 8/64k Request Size

• 100% Random Distribution

• Burst Length = 1

14

MS SQL File Management

• Plan Volume Creation & Data Separation

– Worth the attention - especially with spindle drives

• SQL Best Practices: http://technet.microsoft.com/en-us/library/cc966534.aspx

• Optimize for I/O parallelism

– Stripe size: 64/128/256 KB for SQL 2008+

– Spindles: create multiple files, place on separate physical volumes

• CMDB Data

• CMDB Logs

• TempDB Data

• TempDB Logs

• High IO SSD: can place files on same volume.

– Again, Test the storage subsystem with IOMeter and/or SQLIO

15

Monitoring SQL IO – How to Check IOPs & Latency

• Use Performance Monitor and/or SQL Server Wait Stats

– Latency: < 20 ms (See query)

– Measure IOPs:

• http://sqlserverio.com/2011/02/08/gather-virtual-file-statistics-using-t-sql-tsql2sday-15/

• DiskQLength > 1 or 2

– Can be an indicator

– Can be misleading due to SQL asynchronous IO.

– Watch Average Disk Sec/Read Write & Transfer

• Memory

– Page Life Expectancy: > 300*

– Buffer Cache Hit Ratio: > 95-98%

– Free Pages: > 640

18

MS SQL Settings - Max Degree of Parallelism

• Large and complex queries can over-utilize CPUs

• Default = 0 (change it) – Uses all CPUs

• 2 to 4 on Parent (1x6)

• 1 to 2 on Child

• Never > 8 or # CPUs, whichever is less.

• Use CXPACKET Wait Stats to verify SQL parallelism

• Use wait type query on the next slide

• Or, SSE Tools: http://symantec.com/docs/howto60787

• Consider changing if CXPACKET > 5%

• Increase Cost Threshold for Parallelism to 10

– Reduces chance of simple queries going parallel

19

MS SQL Settings – Memory Configuration

• On x32, use AWE. Don’t use AWE on x64 SQL

– x32 SQL with AWE, /3Gb, /PAE, etc

– AWE is deprecated for x64:

• http://msdn.microsoft.com/en-us/library/ms175581(v=sql.105).aspx

• Enable the option “Optimize for Ad hoc Workloads”

– ITMS has high degree of Ad-hoc/dynamic queries that are non-cacheable

– Doesn’t waste caching on Single-Use queries

– Caches on second+ request

• Set “Maximum Server Memory” value

– In 'Server memory options' page

– leave enough memory for the Operating System &SQL Management Studio to run

• e.g. around 4-6 GB less than the amount of RAM installed.

NSQLRam = NTotalRam – (NOSRam + NAppRam)

20

MS SQL Settings - Other

• Enable Arithmetic Abort in the Server/Instance Properties • Terminates a query when a overflow or divide-by-zero error occurs.

• http://msdn.microsoft.com/en-us/library/ms190306.aspx

• Enable Read Committed Snapshot mode for improved UI

– Improves reporting & console performance without blocking other activity

alter database Symantec_CMDB set read_committed_snapshot on with rollback after 30;

• Disable Autoshrink and Autogrow

– Control Growth Settings by 500 MB (not by %).

– Avoids unnecessary file re-allocations latency

– Avoids unnecessary CPU and IO cycles.

21

SQL Maintenance Plan Recommendations

• Update Statistics (nightly)

– Ensures query processor makes “the best” choices it can given ITMS implementation.

– Or, enable Auto Update Statistics Asynchronously

– Database Properties > Options page.

– !! Enabling affects your ability to put a database into single-user mode (For Maintenance)

• Regular Database Page Maintenance

• Rebuild Indexes (Monthly)

• Reorganize pages (Weekly)

• Regular backup

– Simple Recovery Mode

• Yields Lower IO but you should have an aggressive backup strategy vs. FULL logging with backups

22

Infrastructure & Product Considerations

23

Customizations & Usage

• Evaluate Scheduled Items

– Frequency, number of computers targeted, volume results, Purging/History…

• Tasks & Policies

– Reduce targeting & scheduling, frequency and results

• All types: Custom Inventory, Software, Inventory, Patching, …

• Reports

– Optimize reports that you run often

– Query lowest level resource views in the resource type hierarchy

• Filters

– Optimize all used filters you create

– Query lowest level resource views in the resource type hierarchy

• Inventory Solution

– Limit frequency & Amounts to only what is needed.

24

Product Configuration – The Heavy Hitters…

Setting < 1000 1000 – 5000

5000 – 10,000

10,000 – 15,000

15,000 to 20,000

Agent Configuration Interval Every 1

hour Every 1

hour Every 2 hours

Every 3 hours

Every 4 hours

Full inventory collection schedule Monthly Monthly Monthly Monthly Monthly

Delta inventory collection schedule Weekly Weekly Weekly Weekly Weekly

Complete Resource Membership Update Daily Daily Daily Daily Daily

Delta Resource Membership schedule 15 min 20 min 30 min 45 min 1 hr

Task Service task update interval 5 min 5 min 15 min 15 min 15 min

Policy Refresh schedule 5 min 10 min 15 min 15-20 min 15-20 min

25

Core Settings – Cache Sizes

• Cache size warning in the diagnostics logs: • Source: Altiris.NS.Security.SecurityCache.HandlePermissionCacheFloodedDescription:

Core setting SecurityCacheSize is insufficient to handle incoming security checks. Current Size:xxx

• Bump up cache sizes

– CodeBaseCacheSize: NumPkgs * NumPkgServers

– ItemClassCacheSize: 1000 or Number of rows in Class table

– ItemCacheSize: 3000 to 100,000

– SecurityCacheSize: 5000 to 100,000 (800,000 on full parent)

– StringCacheSize: 1,000,000 or # Rows in String table

– ResxCacheSize: # Strings in StaticString table

– ClientPolicyCacheMaxSize: NumPolicies + Growth of Policies

26

Other Core Settings

• ResourceAssociationTablelockThreshold: 1,000,000

• ReportMaxRowCount from: 50,000 to 500,000

– Generally, set it to 0 unless performance is an issue

– Base on the use case.

• Replication Settings

– ReplicationDataBatchSize: 5,000 – 10,000

– ReplicationEventBatchSize: 5,000 – 10,000

– ItemStateMaxSizeInKB: 128MB

• size of the data container to allow for the increase in the Batch Sizes

27

Other Core Settings - Report Auto Run Control

• SuppressReportAutorun

– When enabled reports do not run immediately.

– Affects every report

– Significant Console Improvement

• When enabled you must click Refresh to get results.

– Disabled by default

28

Enhanced Console View Settings

• Use Saved Searches for frequently used search

• Turn off Search as you type

• Limit result lists

– < 20,000 (consider < 500)

29

Hierarchy Considerations

• Tune it, don’t accept out-of-the-box settings

– Don’t schedule full replication (includes re-replication of full security)

• Do FULL rarely/manually (mostly in troubleshooting)

– Leverage Differential Replication Schedules (first differential is a full)

– Always stagger replications: Child 1 @ 1 am, Child 2 @ 3 am…

• Leave time for success (avoid schedule clashes)

• Also stagger with other heavy activities such as full inventory.

• Limit what gets replicated

– Event replication

• Only data that you must act on.

• Leverage IT Analytics over replication.

– Avoid extraneous replicated resources & data -- ie Software Files

30

A Few more to ponder…

• Change log locations

• Move the event queues

• Don’t compress Event Queue and Replication path

• Use File System junction points to move Replication data folder

• Tune SMP logging & Managed Delivery verbose logging

– Use as needed

• Clear profile trace after each capture, don’t leave it running

• Use history judiciously

– Be cautious what data you keep and the duration that you keep it.

– Consider adding indexes if your history is impacting operations (such as viewing the resource or resource deletion).

31

Operating System & IIS

32

Operating System Tuning

• Optimize Performance for Background Services

• Control OS Background Activities

– Restrict schedules

• Limit AV Scanning – Exclude Folders & file types • *.MDF,*.LDF,*.TMP,*.NSE, Replication & Log Folders

• Services to Consider Disabling

• Indexing Service

• Encryption

• Compression

• AV Scanning

• TEST FIRST!

33

No Internet on SMP? – Disable .NET CRL Validation

• General .Net Assembly functionality

– Published .Net Assemblies DLLs are digitally signed

– Signatures are validated when assemblies are loaded

• Certificate Revocation List validation at crl.microsoft.com

– 30-60 second timeout delay for each assembly load

• ITMS Symptoms

– Windows services on the Platform computer sometimes fail during startup.

– Some Symantec Management Console pages take a very long time to load.

– The SMP may experience performance issues after being offline for 15 days

– Applies to Site Servers w/o Internet also

• Resolution

– Disable CRL lookups

– You can re-enable CRL lookups easily

– Apply to Remote Site Servers

– Via Hosts File, Registry, or Machine.config

Edit the x86 & x64 Machine.config Files:

<configuration>

<runtime>

<generatePublisherEvidence enabled="false"/>

</runtime>

</configuration>

34

IIS & Internet Explorer

• IIS data compression

– Not especially effective since polices are already compressed but, can give some improvements for other activities

• Issues with Kerberos vs NTLM (Frequent Prompting for Credentials)

– http://support.microsoft.com/kb/2749007 (unexpected 401.1 errors)

– Disable IE Pre-Authentication Headers on Machines accessing the console

OR

– Turn Off Kerberos Authentication in IIS

• HTTP Logging (disable or reduce)

35

Topic Links

Planning & Implementation Guide

• http://www.symantec.com/business/support/index?page=content&id=DOC5131

SQL IO Tuning

• http://sqlserverio.com/2011/02/08/gather-virtual-file-statistics-using-t-sql-tsql2sday-15/

• http://www.sqlskills.com/blogs/paul/how-to-examine-io-subsystem-latencies-from-within-sql-server/

• http://blogs.msdn.com/b/askjay/archive/2011/07/08/troubleshooting-slow-disk-i-o-in-sql-server.aspx

IOMeter/SQLIO Tools

• http://www.microsoft.com/en-us/download/details.aspx?id=20163

SSE Tools • http://symantec.com/docs/howto60787

References & Resources

36

Topic Links

Task Server Practices

• http://www.symantec.com/business/support/index?page=content&id=HOWTO48411

Package Staging Practices

• http://www.symantec.com/connect/articles/how-handle-getpackageinfo-request-storms

• http://www.symantec.com/connect/blogs/working-codebasecache-post-getpackageinfo-series

Constrained Pkg Servers

• http://www.symantec.com/business/support/index?page=content&id=HOWTO6907

• http://www.symantec.com/connect/forums/constrained-vs-unconstrained

Package Replication

• http://symantec.com/docs/howto60787

• http://www.symantec.com/business/support/index?page=content&id=HOWTO42304

• http://www.symantec.com/business/support/index?page=content&id=HOWTO44198

References & Resources

37

Embrace Choice

Thank You!

38