ema apm 2012 radarreport summary

7/31/2019 EMA APM 2012 RadarReport Summary

1/29

EMA Radar for Application

Performance Management (APM)

for Cloud Services: Q1 2012

Report Summary

18 Cloud-Ready APM Solutions Available TODAY

By Julie Craig, Research Director

Enterprise Management Associates (EMA)

January 2012


2/29

Table of Contents

2012 Enterprise Management Associates, Inc. All Rights Reserved.

I

EMA Radar for Application Performance Management

(APM) for Cloud Services: Q1 2012 Report Summary

Executive Summary .............................................................................................................................................1Introduction and Methodology .........................................................................................................................2

Vendors Covered by this Report .......................................................................................................................4

APM: Evolving with the Cloud .........................................................................................................................4

The Problem Set: Cloud-specic APM Challenges.......................................................................................5

Management versus Monitoring ........................................................................................................................6

Criteria ....................................................................................................................................................................8

Minimum Criteria ..........................................................................................................................................8

Ideal Criteria ....................................................................................................................................................8

Scoring .................................................................................................................................................................10

Cost Efciency .................................................................................................................................................11Cost Advantage ...........................................................................................................................................12

Product Strength ..............................................................................................................................................12

Architecture and Integration ..................................................................................................................... 12

Functionality.................................................................................................................................................14

Vendor Strength ...............................................................................................................................................15

EMA Radar Maps for Application Performance Management ............................................................... 15

Multi Component/Suite Solutions................................................................................................................. 16

Vendors and Solutions Proled................................................................................................................ 16

General Findings .........................................................................................................................................17

Value Leaders ...............................................................................................................................................18

Strong Value .................................................................................................................................................19

Point Solutions ...................................................................................................................................................20

Vendors and Solutions Proled................................................................................................................ 21

General Findings .........................................................................................................................................21

Value Leaders ...............................................................................................................................................22

Strong Value .................................................................................................................................................22

Exceptional Characteristics: Awards .............................................................................................................. 23

AppDynamics: Best Infrastructure as a Service (IaaS) Automation ................................................ 23

Aternity: Most Comprehensive Endpoint Diagnostics ....................................................................... 24eG Innovations: Best Private Cloud Coverage ......................................................................................24

HP: Most Comprehensive and Real-Time Service Model ..................................................................25

IBM: Best Cloud Vision and Design ....................................................................................................... 25

OPNET: Best Overall Automation ......................................................................................................... 26


3/29


1



Executive SummaryOver the past ten to fteen years, the complexities involved in Application

Performance Management (APM) have multiplied exponentially. Asapplications have moved off monolithic hosting platforms and on to

increasingly tiered and integrated systems, monitoring and managingend-to-end execution have become increasingly problematic.

Widespread adoption of Cloud computing introduces additionalcomplexity and signals a paradigm shift in enterprise computing that isimpacting vendors and customers alike. Cloud is changing the structure

of IT organizations and making inroads on traditional on-premiseapplication hosting models. This, in turn, shapes design decisions andproduct investments by APM vendors.

Consider the following facts:1

The majority of medium-sized to large companies have already embraced private Cloud as a viabledelivery option for business-critical workloads.

Infrastructure as a Service (IaaS), a natural extension of private Cloud, is gaining momentum, with66% of companies either already using IaaS or planning to do so during the next 12 months.

Software as a Service (SaaS) has become a viable alternative as well. While 44% of companies arealready utilizing at least one SaaS service, another 33% plan to do so over the next 12 months.

Companies are deploying Cloud services in some very sophisticated ways. Almost 50% are runningtiered transactions/services that span both Cloud and on-premise. Thirty-ve percent haveintegrated or are in the process of integrating multiple SaaS applications.

Cloud readiness is becoming a factor in developing Requests for Proposals (RFPs). Today, over40% of companies surveyed are re-assessing the Cloud-readiness of their management toolsinvestments for SaaS (35%), Platform as a Service (PaaS- 29%), and IaaS (29%)

Clearly, Cloud adoption is challenging traditional application management models and the APMsolutions that support them. Widespread adoption of private Cloud technology drives newrequirements for visibility to virtual environments. Public Cloud presents unique challenges because,while IT still has responsibility for service delivery, it has virtually no control over the hosting platform.

And while IaaS offers almost unlimited capacity on demand, most IT organizations lack managementsolutions capable of end-to-end management and automated remediation across on-premise andoff-premise Cloud environments. Finally, while the purpose of enterprise management solutions is

to deliver visibility and control, the bottom line impact of Cloud is to obscure visibility and limit thecustomers control.

This EMA Radar explores these challenges in more depth. It focuses on the issues IT organizations areencountering in terms of managing Cloud environmentspublic, private, and hybrid and providesin-depth analysis of solutions that can help solve them.

1 EMA Research on APM for Cloud Services

Cloud is changing the

structure of IT organizationsand making inroads on

traditional on-premise

application hosting

models. This, in turn,

shapes design decisions

and product investments

by APM vendors.


4/29


2



EMA sees this as an early-stage market with vendors adding Cloud-focused products and features at afurious pace. In developing this Radar Report, EMA engaged eighteen top providers of APM solutions

in a detailed analysis of the scope and capabilities of their offerings. These solutions represent arich cross-section of the IT management tools landscape, ranging from small to very large, frompure software to appliance- and SaaS- based, and from point products to extensive multi-component/

multi-function suites. EMAs intent in producing this report is to givebusinesses seeking Cloud-ready APM solutions a starting point fromwhich to develop a short list of potential solutions, based on theirown specic requirements.

Because nearly thirty vendors were offered the opportunity toparticipate in this rigorous study, EMA sees the eighteen vendors who

did participate as the most Cloud-ready in the industry. Although thisis still an early market, these vendors all have distinctive capabilitiesfor managing private, public, or hybrid Cloud environments whichcan deliver signicant value to companies of virtually any size or in

any industry.

Introduction and MethodologyEach vendor completed a comprehensive questionnaire consisting of nearly 100 questions and over500 data points. The survey questions covered the ve key functions common to all EMA RadarReports, which include Architecture, Functionality, Deployment and Administration, Vendor Strength,

and Cost Advantage.

EMA also conducted lengthy interviews and demos with each vendor to clarify product capabilities and

vendor direction. Finally, EMA conducted more than 40 interviews with customers using the products.

While all vendors provided some level of access to their customers (from one to three customers),availability of reference customers was viewed favorably in assessing and validating vendor portfolios.

The degree to which customers were readily provided and available for dialog was one of the manyindicators used for validating APM solutions.

Finally, EMA leveraged ongoing industry dialogs and extensive existing knowledge of the APM spaceto evaluate, consider, and validate each vendors strengths and limitations. These evaluations focus onproviding balanced, consistent insights across all vendors and solutions.

EMA has produced a companion piece which is targeted at presenting and explaining Radar Reportsin general, entitled How to Use the EMA Radar Report.2The goal is to use a combined approach to

quantitatively and qualitatively evaluate providers of solutions in a particular IT management functional

area and presenting their relative differences in a clear, graphical format.

2 Available at: http://www.enterprisemanagement.com/research/asset.php/1715/How-To-Use-The-EMA-Radar-Report,

EMA, April 2010.

Because nearly thirty

vendors were offered the

opportunity to participate

in this rigorous study,

EMA sees the eighteen

vendors who did participate

as the most Cloud-ready in the industry.
http://www.enterprisemanagement.com/research/asset.php/1715/How-To-Use-The-EMA-Radar-Reporthttp://www.enterprisemanagement.com/research/asset.php/1715/How-To-Use-The-EMA-Radar-Report


5/29


3



Figure 1: The EMA Radar is optimized to show how vendor solutions cluster in terms of two primary axes: Vendor Strength (architecture,

integration, functionality) and Cost Efciency (ease of administration, deployment, support & services, costs advantage)

Quoting from How to Use the EMA Radar Report, No analysis of this type can tell you which vendor

is best for you. The data collected for an EMA Radar Report can certainly be used to make thatdetermination, but it must be applied to the specics of your current environment, level of maturity,and goals and priorities. Since the authors of any given Radar Report do not have your unique specics,

the Radar Report can only be a starting place and a guideline. It can inform you of the market andshort-cut your process to developing a short list.

Figure 2: Radars for each vendor solution are included in the full report and show a ve-

axis contrast between the average prole and the vendor in question.

Functionality

Deployment &Administration

Cost AdvantageVendor Strength

Architecture &Integration

Specialized Product Low Cost

Functionality

Deployment &Administration

Cost AdvantageVendor Strength

Architecture &Integration

Strong Product Higher Price


6/29


4



Vendors Covered by this ReportAPM solution providers covered in the report include AppDynamics, AppFirst, Aternity, CA,

Compuware, Correlsense, eG Innovations, HP, IBM, INETCO, Nastel, Netuitive, New Relic,OPNET, OpTier, Quest, SolarWinds and Splunk. These solutions are clearly different in multiple

respects, and represent a variety of capabilities, technology vantage points, and form factors. Whilethis differentiation makes a strict apples to apples comparison virtually impossible, it is possible tocompare and highlight features, costs, and overall value proposition relevant to the APM discipline.

In the interest of fairness and for reader clarity, nal scoring has been divided into two broad productgroups based on general architecture. Both groups answered the same questions and were scored

similarly. However, the two groups are charted and averaged separately for this report, dependingon whether the solution is built around a single component or around a suite of pre-integratedcomponents. The net impact is to highlight the strengths of both groups while comparing breadth offunctionality and value proposition against a more comparable group of solutions.

Solution groupings are as follows:

Point Solutions:AppDynamics, AppFirst, Aternity, INETCO, Netuitive, New Relic, and Splunk.

Multi Component/Suite solutions:CA, Compuware, Correlsense, eG Innovations, HP, IBM, Nastel,OPNET, OpTier, Quest, and SolarWinds.

Point Solutions:In general, the point solutions proled here address a targeted functional APM capabilityor set of capabilities.

Multi-Component/Suite Solutions:These solutions vary considerably in terms of capabilities, with CA,Compuware, HP, IBM, Nastel, OPNET, OpTier, and Quest each delivering distinctive takes onintegrated APM toolsets. In contrast, Correlsense and eG Innovations both deliver depth of visibility

addressing specic APM problems via a multi-component form factor. Finally, SolarWinds delivers acost-effective platform for mid-market APM.

EMA awards a limited number of citations to vendors with particularly noteworthy capabilities inone or more functional areas. While the vendors participating in this particular study were all strongand the choice was difcult, only six solutions received awards. Four were chosen from the Multi-

Component group, and two from the Point Solutions group, commensurate with the number ofparticipants in each.

APM: Evolving with the CloudFrom multiple perspectives, APM solutions for Cloud services are still evolving. The ultimate answersto many of the questions related to managing Cloud applications are still being addressed. For example,

interviews conducted as part of this study indicate that many public Cloud providers do not yet offermonitoring agents or APIs, a factor that hampers the capabilities management vendors can build intoAPM solutions. Similarly, virtualization vendors have exposed varying levels of insight into privateCloud execution with some stronger than others. VMware, for example, is a poster child for providing

exceptional manageability capabilities with vCenter and related APIs.


7/29


5



While neither public Cloud nor APM vendors have all the pieces inplace at present, virtually every vendor is aware of the necessity for

developing such capabilities. In fact, many of the vendors included inthis study announced new products and upgrades to existing solutionsduring the short time span required to produce this report.

There have also been signicant developments with non-participatingvendors during this time frame. While BMC declined to participate

because product development was in-process, it has since announcedenhancements to its APM solutions. In short, although this marketis still evolving, it is extremely dynamic and constantly improving.Meanwhile, the management capabilities which are available are already

quite impressive.

The Problem Set: Cloud-specic APM ChallengesTodays applications and transactions can be delivered in a wide variety of hosting environmentsThey can be hosted on-premise via traditional tiered hosting or virtualized private Cloud. They can becomponentized such that elements of the execution software are hosted on-premise and in the public

Cloud. They can access partner and provider Clouds, be delivered via SaaS, and can even encompassSaaS-to SaaS integrations.

While this diversity offers a wide variety of deployment options, it also introduces a bewildering arrayof budgeting, management, and tooling considerations. Some of the most challenging include:

New requirements driving APM purchasing:Many companies are nding that Cloud-relatedAPM requirements are beyond the capabilities of incumbent solutions, and are re-examining

budgets (and tools strategy) accordingly. In other words, existing APM investments arent Cloudready, and cannot support this diversity of hosting environments. From this perspective, Cloud

computing is driving new requirements which are, in turn, impacting budget line items andpurchasing decisions.

End-to-End visibility becomes a must have:Companies that are less operationally mature

are still transitioning from monitoring infrastructure to monitoring/managing applications. Theylack top down monitoring solutions that deliver insight into end-to-end performance. This canput them at a disadvantage in terms of competitive differentiation.

Exponential increase in metrics: Companies leveraging Cloud tell EMA that it introducesnew requirements for metrics and big data processing. Outlier measurements such as TCPthresholds, rates, and queue limits have become important elements of the APM equation, and are

pushing the processing and analytical limits of existing solutions. As an example, one customerinterviewed as part of this research reports that monitoring systems are processing over one billionmetrics daily.

Real time management becomes the new normal: As application systems become

increasingly dynamic, real time analytics capabilities become a must have. This requirementis becoming increasingly pressing as more companies begin to leverage dynamic capabilities suchas VMware vMotions and Cloud bursting to IaaS. It permeates every aspect of APM from

discovery/ dependency mapping to topology visualization and End User Experience Management.

Although this market is stillevolving, it is extremely

dynamic and constantly

improving. Meanwhile, the

management capabilities

which are available are

already quite impressive.


8/29


6



Troubleshooting is more complex: This is probably the biggest challenge, as both public andprivate Cloud deployments increase the number and complexity of moving parts comprising

the application ecosystem. To compound the problem, most public Cloud providers are still inthe early stages of developing management-level instrumentation that enables them to share keyperformance information with APM solutions. At the same time, ITs responsibility is still to the

user, and it is up to IT to diagnose performance problems regardless of where (and how) theapplication is hosted.

For example, while poor SaaS performance can denitely be on the public Cloud provider side, it

can also be impacted by a multiplicity of additional factors such as:

Lack of bandwidth on the Wide Area or Corporate network

Contention on the users desktop

Slow system calls to internal or external integrations

Failure of optimization or load-balancing components Contention with other applications or users

Database issues

Location-based issues

Etc.

Without adequate tooling, knowing who to call is a best guess exercise.

Management versus MonitoringWhile the focus of this research is on Application Performance Management, the term means different

things to different constituencies. This section claries EMAs denitions, which in turn clarieshow vendors participating in this study were assessed. These denitions should also help enable IT

organizations to hone in on the products that best meet specic internal monitoring and managementrequirements.

Application Management: The role of Application Management is to deliver visibility andcontrol for transactions, applications, and services. While Application Management covers avery broad range of capabilities required to monitor AND manage applications (see Figure 3),

Application Performance Monitoring and Application Performance Management are subsets withvarying levels of visibility:

Application Performance Monitoring:Assesses performance and availability of a transaction,application or service

Application Performance Management (APM): Encompasses the functions of performancemonitoring but also supports troubleshooting, problem determination, and root cause analysis.Ideally, and this is an emerging capability, APM also includes the ability to recognize (and

potentially resolve) performance and availability problems BEFORE users are impacted. Inaddition, APM supports and links to other IT functions such as Service Level Management andConguration Management.

Without adequate tooling,

knowing who to call is

a best guess exercise.


9/29


7



Such distinctions are particularly important when it comes to Cloud. While multiple solutions mayclaim to support applications running in public IaaS, SaaS, or PaaS, their levels of granularity can vary

radically. For example, while running synthetic transactions every ve minutes monitors performanceand availability (intermittently), it does not provide the granularity necessary to troubleshoot anddiagnose performance problems, particularly in complex multi-tier environments.

Vendors have approached these challenges from multiple perspectives. So called top down solutionssuch as synthetic transactions deliver a point-in-time perspective which is broad versus deep. Bottom

up solutions provide deep visibility but lack the cross-technology, end-to-end perspective that isnecessary to manage applications versus infrastructure. Clearly, a meld of the two is the holy grail ofAPM, and APM software vendors have made signicant strides in delivering on this vision during thelast one to two years.

Figure 3: EMA End-to-End Application Management Semantic Model

Ultimately, it is essential for Cloud and enterprise management vendors alike to become Cloud-ready.

In support of this idea, EMA has added a Cloud Analytics layer to its End-to-End ApplicationManagement Semantic Model.

This model depicts the management capabilities required to gather and analyze the informationnecessary to track and manage applications from a 360-degree, end-to-end perspective. It includesmetrics gathering and supporting analytics functionality. Although solutions monitoring each of theblue layers in the diagram can be deployed separately (and often are), the collective value of the stack

far exceeds that of each separate capability. Solutions capable of gathering and analyzing metrics frommultiple layers yield a uniquely comprehensive perspective of the application ecosystem. This is criticalto managing, versus simply monitoring, application environments in the Cloud-extended enterprisewhich is a reason why solutions with breadth of coverage across multiple layers of the model achieved

higher scores for this Radar.


10/29


8



CriteriaTo clarify vendor placement in this report, its important to understand the models used to include and

evaluate this very heterogeneous group of vendors. This section species both the minimum criteriafor inclusion and the ideal combination of capabilities used to evaluate all solutions. These criteria

were used as the basis for developing and scoring the survey-based portion of this assessment.

Minimum Criteria1. Product ts within at least one of the layers/categories identied in the Semantic Model or is a

multi-component solution covering multiple categories

2. Product also supports monitoring/management of at least one of:

a) Public Cloud

b) Private Cloud

c) Hybrid Cloud3. All elements of the solution were generally available for purchase (GA) at the time the survey was

conducted (October-November, 2011)

4. At least one customer available for interview.

Ideal Criteria1. Meets minimal criteria listed above

2. Automated versus manual operation: In general, higher levels of automation and minimalreliance on manual tasks received better scores. Examples include optimal (versus excessive)dependence on scripts or scripting, auto-generated and maintained baselines /thresholds, single

agent versus multiple agents, self-learning capabilities, etc.

3. Topology models (or service models): Products supporting auto-generated topology modelsvia application discovery, execution analytics, and dependency mapping were favored over those

requiring manual topology modeling.

4. End to End transaction/application visibility spanning on-premise and public Cloud

5. CMDB/CMS support:Although some of the vendors participating in this research contested this

requirement, EMA research has found that Cloud computing makes CMDB/CMS systems evenmore important. Ultimately, keeping track of where application components are hosted requiresfederated or API-level integration of APM solutions with CMDB/CMS systems. Ensuring that

CMS systems are accurate requires automated dependency mapping. For these reasons, productswith native integration or API support with CMDB/CMS systems (and with dependency mappingfunctionality) were favored over those that did not.

6. Open Integration: Products with native support for integration with third party solutionsreceived higher scores than those that do not. Examples include trouble ticket generation, metricsassimilation and correlation, and visibility to a variety of heterogeneous platforms and solutions.

7. Visibility into public Cloud platforms: The ideal APM solution demonstrates broad and deep

coverage for AND formalized management partnerships with public Cloud IaaS, SaaS, and/or PaaS vendors. Vendors with actual versus black box visibility into public Cloud platforms,typically via API or similar integration technology, received higher scores.


11/29


9



8. Visibility into virtualization platforms: Support for private Cloud and IaaS also requires broadand deep coverage of virtualization platforms. Again, this depth of visibility requires integration

with management capabilities delivered by the virtualization vendor.9. Above-average investments in Research and Development

10. Support for hybrid Cloud, including visibility to integration points

11. Customer Support: Multiple customer feedback options, support options, and interactionmethodologies (user groups, online forums, conferences, training, etc.) enabling customers to

easily communicate with one another and with the vendor.

12. Fast time to value: While time to value is generally easier to achieve with point solutions versussuite solutions, vendors of all sizes are investing in making this process as easy as possible.

Distributing solutions as pre-loaded hard or soft appliances, delivering analytics functions as SaaS,and making application templates and ngerprints available are examples of vendor efforts thatimprove time to value.

13. Price commensurate with capabilities:While price is certainly a concern, higher value solutionsrequire higher levels of vendor investment and therefore cost more. EMA focuses on CostEfciency versus cost alone, a measurement that takes overall value, product and deploymentcosts, training and support costs, and ongoing administration and consulting costs into account.

14. Proven scalability

15. Customer interviews: While at least one customer interview was a requirement for each vendor,

additional customers positively impacted overall scoring.


12/29


10



ScoringFor all Radar Reports, EMA evaluates solutions based on ve key areas represented by the ve sides of a

hexagram. They include Deployment and Administration, Cost Advantage, Architecture and Integration,Functionality, and Vendor Strength. The last category, perhaps the only one thats not self-explanatory, is

focused on market and industry presence, vision, and the nancial stability of the vendor.

Within each of these ve evaluation areas, EMA creates a superset of capabilities spanning the

known solutions in the marketplace. For each superset, specic Radar Reports add questions aboutnew and emerging areas (in this case virtualization and Cloud), and balance the results with standardcomparators used across all EMA Radars. The evaluation model used for this Report is presented as

Figure 4. The following section details each evaluation area, along with its scope and rating priorities.

Figure 4: Assessment Model for APM/Cloud Radar


13/29


11



Cost EfciencyThis is the rst set of measures within the EMA Radar Report framework and one of the two

major axes of the Radar distribution diagram. It consists of two major sub-sections: Deployment &Administration and Cost Advantage. Each of these, along with the breakout capabilities assessed for

each, is described below.

Deployment & Administration

Deploying and administering is the rst side of the hexagram. These scores include measures thatindicate how easy or difcult it is to deploy a given APM solution into the production environment, as

well as initial time to value. The sub-areas assessed include:

1. Ease of Deployment: Resources required to initially deploy the product. High marks go to

fast time to value, vendor-provided quick start aids (such as templates), minimal consultingrequirements, and support for a variety of databases and database vendors. Simplied packaging

such as a SaaS form factor also wins points.a) Time to deploy:Vendors were asked to gauge their answers for a hypothetical customer

with two multi-tiered enterprise applications. Within this context, specic questions addressedthe average time required to deploy, begin data collection, and deliver functional, actionableperformance data. Additional questions addressed the professional services requirements and

overall costs associated with deployment.

b) Packaging requirements and options: Operating environment requirements and availableform factors. Scoring was based on whether the solution includes hardware and software as

part of the quoted cost, is turnkey or requires add-ons, is available as SaaS (since SaaS-basedproducts are obviously easier to deploy than on-premise solutions).

c) Staff training: The vendors training offerings. Scoring favored breadth of options (i.e.,

online, partner led, on-premise, etc.) and lower overall cost. It also factored in the averagenumber of training hours necessary for basic prociency and administrator level efciency,since training costs for multiple personnel can add signicant costs.

d) Disruption minimization:The impact to normal IT operations that occurs during productdeployment and/or upgrades.

2. Support and Services: Assesses the vendors customer support, professional services, and

product maintenance options. High marks go to vendors with a wide variety of options to supportcustomers of different sizes and with different budgets.

a) Customer Support: Vendors are scored on the breadth of support options (online, business

hours, 24 X 7, etc.). Also takes into account support levels and vendor-provided vehicles

for customer product input (product suggestions and status for public review), as well asavailability of user interaction options (online, conferences, etc.).

b) Professional Services: Professional services options, whether vendor or third-party delivered.Higher scores go to vendors that include professional services as part of the quoted solutioncost (to eliminate unpleasant surprises) and to solutions that require minimal professional

services to deploy.

c) Version Maintenance: Frequency of full releases and software upgrades.


14/29


12



3. Ease of Administration: Assesses the product and personnel costs necessary to congure,administer, and maintain the solution on an ongoing basis.

a) Ongoing administration: The manpower and time required to manage the solution.b) Update Process:The operational impact and time requirements for upgrades, as well as

ease-of-use features that support it.

c) Testing/Migration: License fee costs for licenses utilized for product upgrade pre-testing.

Cost AdvantageCost Advantage is the second side of the hexagram, and assesses overall cost factors related to Pricing,Licensing and ongoing maintenance.

1. Price: Average license cost for a typical deployment, including whether additional for paymodules are necessary, and whether consulting and/or training hours are included in the quoted

cost. Also assesses the time required for the average customer to recoup full ROI.

2. Licensing Models: Breadth of licensing models and form factors. Higher scores go to solutionsavailable via multiple licensing models and form factors, which makes solutions more readilyavailable to a wide variety of company sizes and budgets.

3. Maintenance Costs:Average maintenance fees for highest level of service. Industry average isapproximately 1922%; higher scores went to solutions with lower maintenance fees.

Product StrengthProduct strength is the second major axis of the Radar distribution diagram, and is comprised of twosub-areas, Architecture/Integration and overall Functionality.

Architecture and IntegrationThe rst of the two major product strength categories is Architecture & Integration. It is the third sideof the hexagram, and assesses the enabling technology underlying the APM functionality.

1. Architecture:Assesses design elements contributing to overall functionality and usability.

a) Design: Design characteristics such as deployment form factors (SaaS, SaaS with on-premisecollector, VM, etc.) and metrics collected. Examples include:

i. Data collection methodologies (Synthetic transactions, EUE, RUM, tracking/stitching,

instrumentation points, platform coverage)

ii. Supplemental data sources monitored (Protocols, network ow, third-party agents,

CMDB/CMS, discovery sources, etc.)b) Scalability: Scalability is a key consideration, particularly for rapidly growing companies

and large enterprises. Specic questions measured the products ability to scale across large,geographically distributed environments and the largest deployment to date.


15/29


13



c) Breadth of Environments Supported

i. Support for public, private, and hybrid Cloud environments

ii. Support for a variety of tiered transaction environments, i.e. on-premise/public Cloud,on-premise distributed/mainframe (often part of Cloud deployments), SaaS-SaaS

integrations, etc.

d) Breadth of Applications Supported:

i. Application deployment types supported, i.e. Web-based, fat client, Web 2.0. SOA,

Mobile, VoIP, etc.

ii. Technologies supported, i.e., ESB/EAI, API-based integrations, Middleware, Storage,CDN.

iii. SaaS and IaaS partnerships, which indicate the depth of support for public Cloud vendors(SaaS vendors such as Salesforce.com and OpSource and for IaaS vendors such as

Amazon and Rackspace). Higher scores went to APM vendors who have built visibilityto execution within public Clouds, such as EC2 and Salesforce.com, versus those simplyleveraging synthetic transactions as the primary Cloud monitoring technology OR thosemonitoring public Cloud as a black box.

iv. Specic support for and partnerships with virtualization vendors supporting privateCloud. Higher scores went to APM vendors who have built depth of visibility to executionwithin private Clouds, and those who have incorporated technologies such as vMotions

and vCenter integration into end-to-end monitoring and management dashboards andcorrelation.

2. Integration/Interoperability: Virtually every IT organization has a variety of enterprise

management solutions and, sooner or later, will need to integrate. This section assesses the ability

of the solution to integrate with other products within the vendors portfolio, as well as with thirdparty solutions.

a) Open Integrations: Integrations to generic events such as SNMP traps as well as to vendor-delivered APIs for event, data, and Cloud integration.

b) Event Management Integration: The most common fault/event management platformssupported, and whether APIs that support these integrations are provided.

c) CMDB/DMS Integration:Several participating vendors questioned whether CMDB/CMSintegration is important for APM solutions. EMAs position is based on survey-based data,which indicates that IT organizations are starting to question how Cloud-based services can

be managed and tracked. They see CMDB/CMS solutions as the ultimate answer for tracking

component sources for increasingly complex and interconnected services.


16/29


14



FunctionalityThe second product strength category is Functionality. It is the fourth side of the hexagram, and

covers product features and ease of use.1. Features: Features are the heart of any solution, and the overall value proposition ultimately boils

down to what a solution can actually do. Features were comprehensively assessed across sevenkey categories and scoring favored features supporting automation versus manual conguration,

modeling, and analytics. Key areas assessed include:

a) Application discovery/recognition: One key feature EMA sees as being fundamental tomanaging complex applications is a visual service topology model, ideally generated in real

time and requiring minimal (or no) manual input. Application discovery and transaction tracingare key aspects of generating such a model. Assessment questions covered the breadth ofelements leveraged to identify a given application (i.e. port, protocol, IP address, ngerprintsor templates, packet analysis, etc.) and available discovery methodologies.

b) Metrics and measurements: Metrics and measurements delivered natively and via integratedthird party solutions. Examples include performance by tier, performance juxtaposed withinfrastructure (topology), performance by user/group, completed versus uncompleted

transactions, etc.

c) Cross-technology visibility: Functionality related to tracing transactions/applications/services across heterogeneous tiers and across hybrid on-premise/public Cloud environments.

d) Alerting/alarming: Automated alerting/alarming features and triggers. Examples includetrouble ticket generation/clearing, dynamic baselines, automatic threshold setting, and dynamicvariable thresholds by hour, day of the week, etc.

e) Troubleshooting:Features related to improving Mean Time to Resolution/Recovery (MTTR)

such as visual triage, probable cause analyses and rankings, time-based metrics comparisons,performance comparisons by users/groups/geographies, etc. Assessment covers general

APM, private Cloud, SaaS applications, and IaaS applications/components.

f) Security and user management: Security-related features such as single sign-on and LDAPintegration, user role support, and access control.

g) Analytics/advanced analytics: In this context, advanced analytics generally covers featuresspecically designed to maximize automation and minimize manual interaction with the toolitself. Specic features assessed include self learning of the application environment, self

conguration of the solution based on environmental awareness, rolling baselines thatself-update based on ongoing trending, self learning of normal performance across hops,automated real-time service modeling, and visibility across variable transaction execution

paths.2. Ease of Use: Ease of use can make the difference between a product that is fully utilized and one

that becomes shelfware. If a product is easy to use, it will require less training and less consultingover time and can be used by personnel with varying levels and types of skills.

a) Visualization/reporting: Mapping, grouping, visualization, and reporting capabilities. Theideal solution will include a mix of frequently used canned reports, support for ad hocreports, and a facility for data sharing (export and import) with tools such as Microsoft Excel.

b) Roles supported:Assesses most common IT roles utilizing the solution.


17/29


15



Vendor StrengthThis is the fth and nal side of the hexagram and covers the multiplicity of factors underlying overall

vendor strength. Collectively, the measurements in this section determine the size of the bubbleon the chart. These factors are meant to gauge the overall viability of an APM provider, as well as

the quality of their vision, strategy, product execution, and market voice. It consists of the followingspecic categories:

1. Vision:Vision becomes innovation, and visionary vendors create products that enable customersto more easily assimilate changing technologies and application architectures. For this assessment,vendors were asked to describe their Cloud-related product strategy and vision for APM in 500

words or less. More visionary vendors received higher scores.

2. Strategy:Strategy enables vendors to achieve their vision. Again, vendors were asked to describetheir longer-term strategy for advancing APM capabilities.

3. Financial Strength: Components of this score include the vendors own assessment of nancialposition, and estimated year over year revenue growth for the APM solution. For public companies,it also incorporates data included in public documents, such as cash available and forward P/E

estimates.

4. Research and Development:Vendors were asked to disclose R&D investments via a multiplechoice question with ve potential answers: 0 5%, 6 10%, 11 15%, 16 20%, and 21 % +.

This answer was then ranked against the answers of other vendors in the study and entered as ascore on a sliding scale between 1 and 10.

5. Market Credibility: While features and functions are critical, vendor credibility is another keyaspect of product selection which goes far beyond marketing. In fact, for assessments of this

nature, marketing hype often does more harm than good, as true product capabilities, strengths,

and weaknesses can be obscured by market speak geared toward misinformation versus reality.In evaluating credibility, EMA examined a number of measures, including in-depth interviews

with customers. In general, vendors providing two or more customer references were viewed morepositively than those providing fewer. Other relevant factors include: how focused the vendor is onthe APM space, how straightforward the vendor was in terms of the assessment, how transparentthe vendor is to the analyst community, and treatment by other credible industry voices.

EMA Radar Maps for Application Performance

ManagementAs indicated above, nal scoring has been divided into two broad groups by basic high-level architecture.The results for the two groups are presented separately, beginning with the Multi Component group.

As a review, the vendors in the two groups include:

Multi Component/Suite solutions:CA, Compuware, Correlsense, eG Innovations, HP, IBM, Nastel,

OPNET, OpTier, Quest, and SolarWinds

Point solutions:AppDynamics, AppFirst, Aternity, INETCO, Netuitive, New Relic, and Splunk


18/29


16



Multi Component/Suite Solutions

Figure 5: EMA Radar Map for APM: Multi Component/Suite Solutions

Vendors and Solutions Proled CA: CA APM

Compuware: Compuware Gomez Application Performance Management

Correlsense: Correlsense SharePath

eG Innovations: eG Enterprise Suite

HP: HP APM

IBM: IBM Tivoli Application Performance Management Solution

Nastel: Nastel AutoPilot

OPNET: OPNET APM Xpert Suite

OpTier: OpTier CloudFirst (part of OpTier BTM suite)

Quest: Quest Foglight

SolarWinds: SolarWinds Application Performance Monitor, SolarWinds Synthetic End User Monitor


19/29


17



The value of a solution is dened by comparing its strength with its cost effectiveness. Figure 5 providesa graphical representation of the industry leaders evaluated by this study, positioned in relation to both

critical axes. Scores on the x axis reect Cost Efciency, which EMA considers to be a measure ofoverall value. Products shown towards the right on this axis tend to be less expensive or have strongfeature/function in relation to overall cost. The y axis reects the vendors coverage of the specic

functionality assessedin this case, APM for Cloud services. The size of the bubble represents arelative measure of Vendor Strength.

General FindingsAll of the vendors in this category scored as Strong Value or better. This indicates that all exceededEMAs rigorous scoring criteria. Although this may appear to show an overly rosy judgment on EMAspart, the truth is rooted in the phenomenon of self-selection. Vendors not prepared to invest theresources required to traverse the Radar process, those daunted by the rigor of the survey and dialogue,

and those less certain of product strength declined to participate. From this perspective, those who did

participate can be considered the APM elite those providers invested in delivering APM solutionswith substantial Cloud-focused value to application management practitioners.

HP and IBM were top scorers in the Value Leader quadrant, although a surprising outcome of thisstudy is that four additional vendors are positioned in this quadrant as well. IBM and HP have both

focused on closing the pricing gap in recent years, HP with increasing focus on a SaaS form factorfor some capabilities and IBM with the introduction of hard and soft appliances (and, most recently,a SaaS option as well). eG Innovations and Nastel are both known as lower cost APM alternatives,with Nastel in particular focused on delivering rich APM functionality at less cost than traditional Big

Four competitors. eGs strong suit is private Cloud management, with very strong support for a widevariety of virtualization vendors. Compuware, of course, has been a leading APM vendor for quitesome time, and the acquisition of Gomez cemented its place as a high-value APM solution. Finally,

OPNET is a more recent entry into the APM market, led by high levels of automation and deepvisibility to ow-based messaging.

CA, Quest, OpTier, Correlsense, and SolarWinds all placed in the Strong Value category. CAs

mainframe support and vendor-agnostic stance make it a very attractive option for companies seekingto consolidate metrics from multiple third party solutions and thereby maximize existing investments.Quest has a very strong contender in Foglight, with a loyal customer following based on ease of

installation and maintenance as well as breadth of functionality3. OpTier is in a class of its own in termsof Business Transaction Management, and has recently announced Always-on APM, a methodologyfor delivering ongoing transaction tracing with minimal impact to production platforms. Correlsenseoffers an exceptional End User Experience Management solution, which is distinctive in that it can be

used to monitor both internal and external users. Finally, SolarWinds, known primarily as an enterprise

management platform for the mid-market, continues to add application-focused features that make itincreasingly competitive in the APM space.

In-depth proles for all vendors in this report are listed in alphabetical order in the APM VendorProles section.

3 EMA has documented signicant Return on Investment by Quest customers in related studies


20/29


18



Value LeadersCompuware: Compuware has invested heavily in Cloud management technology, and the Gomez

Application Performance Management (APM) solution bundle covers a broad array of on-premisesand SaaS-delivered solutions. Perhaps best known for its agship Compuware Gomez Data Center Real

User Monitoring (RUM) product (formerly Vantage), Compuwares Cloud solutions are a result of itsacquisition and successful assimilation of two key technology companies. The acquisition of Gomez in2009 brought with it rich synthetic and real user monitoring capabilities for Internet-delivered servicesfrom the outside in via a global network of monitoring stations. The acquisition of dynaTrace in 2011

added the inside out, code level perspective. These acquisitions substantially improved CompuwaresCloud (and APM) coverage, which now combines metrics from both external and internal monitoringwith cross-technology visibility via the Infrastructure and Component Analysis solution.

eG Innovations: While eG Innovations eG Enterprise Suite is a well rounded transaction monitoringand management solution, eG is particularly strong in terms of its private Cloud coverage. eG has

invested heavily in building support for virtualization, with its trademark In-N-Out virtual infrastructuremonitoring supporting performance troubleshooting with both depth and breadth of applicationvisibility. A key differentiator is eGs single monitoring agent architecture (covering a wide variety ofheterogeneous technologies), which can considerably decrease administrative overhead over time.

HP: HP APM is a robust solution, which is particularly noteworthy for its troubleshooting capabilities.HP has invested heavily in developing a Run Time Service Model (RTSM) as an overlay to the HP

Universal Conguration Management Database (UCMDB). Leveraging metrics harvested from acrossthe execution ecosystem, HTTP\RMI tagging, and cross-VM correlations, HP discovers relationshipsamong application components and reports them to RTSM. This delivers a central, up-to-date modelof the application run time architecture, which in turn provides a data source for a Federated CMDB/

CMS. EMA sees this as a visionary approach to delivering run-time topology, which is essential to

managing, versus monitoring, applications in Cloud environments.IBM: IBM Tivoli Application Performance Management Solution includes ve component solutionswith deep and broad coverage for public, private, and hybrid Cloud. Since IBM relies on its partnerchannel to sell APM solutions to the mid-market, IBMs customers for this solution are primarily

within enterprise and Service Provider markets.

IBM Softwares Tivoli solutions encompass enterprise management for production IT (and extended

business technology) monitoring. One very noteworthy differentiator is the fact that IBM has notonly tightly integrated its pre-deployment (i.e., Rational) and post-deployment (Tivoli) solutions, it hasactively reused components across the two lines. This delivers two primary benets to customers. Bycontrolling development costs, IBM can pass on savings. In addition, products leveraged by different

IT constituencies have a common look and feel. This smooths the transition to cross-functional

DevOps by reducing the learning curve required for DevOps teams. Longer term, it also paves theway for extended DevOps, as applications increasingly span public and private Cloud.

IBM Tivoli APM focuses on predictive analysis, capacity management, and proactive problem isolation.The packaging strategy is to deliver a comprehensive set of capabilities as modular extensions on asingle, scalable and comprehensive product infrastructure while preserving the ability for customers to

easily integrate third party applications and data. Tivoli specically addresses public Cloud by providingintegration capabilities to securely monitor and provision across hybrid Cloud environments.


21/29


19



Nastel: Nastel AutoPilot is essentially a top level data-agnostic aggregator, capable of assimilatingand analyzing data from Nastels own collectors as well as from a wide variety of third party sources

and business data. The Nastel Complex Event Processing (CEP) engine forms the core platform ofAutoPilot and is the heart of this solution and a distinctive differentiator. Nastel is exceptional in itscoverage of data-center-based technologies, and previous EMA research attests to impressive ROI

in such environments. AutoPilots functionality is driven by extensive self-learning capabilities andsophisticated data aggregation analytics.

OPNET: OPNET APM Xpert Suite is a comprehensive solution for managing the performance ofapplications in physical, virtual, and Cloud environments. OPNETs High Denition PerformanceManagement approach combines broad visibility across domains with granular drill-down within eachdomain. It heavily leverages analytics to automate troubleshooting and root cause diagnosis, minimizing

the manual conguration, topology modeling, and scripting overhead necessary for many competingAPM solutions. EMA sees OPNET as a good example of a new breed of solutions which automatethe process of knitting together multiple information sources, including network ow, into real timemodels of application execution-- with minimal manual intervention.

Strong ValueCA Technologies: CA Technologies CA APM suite monitors user transactions in real time as they

traverse the hybrid-Cloud infrastructure (physical, virtual, public Cloud) and is designed to proactivelyidentify, diagnose, and resolve application-related issues before end users are affected. CAs vendoragnostic design supports integration with a wide variety of third party solutions. In addition, excellentvirtualization support gives customers a multiplicity of private Cloud options, as CA is one of few

vendors covering Microsoft, Citrix, Red Hat, VMware, and Oracle. This solution also boasts strongsecurity and reporting features.

Correlsense: Correlsense is squarely positioned in the End User Experience (EUE) market, with fullyintegrated capabilities supporting Application Performance and Business Transaction Management.Correlsense SharePath uses a lightweight, no touch approach to accurately trace transactions acrosscomplex, multi-tier dynamic architectures without modifying application code or tagging payloads.

This provides in-depth visibility and analytics for transaction performance in cloud environments. Inaddition, Correlsense SharePath RUM gathers deep dive user experience metrics via agents and asmall snippet of code downloaded into the user browser. Because of this design, it can monitor End

User Experience for either internal or external users, a core strength that lends itself to a broad rangeof use cases. Correlsense integrates with CMDB/CMS systems and is one of few vendors with specicsupport for SaaS vendors, including Salesforce.com, SugarCRM, and Google.

OpTier: OpTiers agship solution, OpTier BTM, is a highly automated transaction managementsolution. OpTier cut its teeth in the nancial services vertical to monitor mission-critical trading and

nancial transactions. One distinctive differentiator is OpTiers ability not only to monitor application-related problems, but to take action to mitigate them as well. OpTier CloudFirst is a fully-integratedcomponent of OpTier BTM designed for business-centric Cloud transaction monitoring. CloudFirstdelivers visibility into transactions that ow in and out of public and private Clouds, and includes a setof templates to help customers assess the Cloud-readiness of applications before deployment. OpTier

is exceptional in its coverage of on-premise-hosted technologies, and previous EMA research atteststo impressive ROI in managing large-scale data center environments.


22/29


20



Quest:Traditionally known for its richly functional database management solutions, Quest successfullytransitioned to the Transaction Management market with its acquisition of Foglight Software in 1999.

During the 2006-2007 timeframe, Quest re-engineered Foglight as a modular, hub and spoke solutionfeaturing an analytics and processing core and multiple snap-in components. This model has enabledQuest to adapt to industry disruptions such as Service Oriented Architecture (SOA) and Cloud more

nimbly than some competitors. To address Cloud, Quest combines a variety of modular productsaddressing EUE, infrastructure, database, server, virtualization, and network, all of which seamlesslyintegrate to the Foglight Performance Monitoring platform.

SolarWinds: SolarWinds Application Performance Monitor (APM) and Synthetic End User Monitor(SEUM) solutions offer a wide range of affordable APM capabilities. SolarWinds is popular with ITprofessionals who regard it as a cost-effective and scalable enterprise management option. SolarWinds

is investing in building its APM functionality with coverage for an array of application types includingweb-based, fat client, Web 2.0, SOA and VoIP (with add-on software modules). It is also noteworthyfor its comprehensive security capabilities, such as single sign on, custom or out-of-box user credentialsby role, and the ability to issue credentials via software.

Point Solutions

Figure 6: EMA Radar Map for APM: Point Solutions


23/29


21



Vendors and Solutions Proled AppDynamics: AppDynamics Pro

AppFirst: AppFirst

Aternity: Aternity Frontline Performance Intelligence Platform Version 5.0

INETCO: INETCO Insight

Netuitive: Netuitive 5.5

New Relic: New Relic Pro

Splunk: Splunk Enterprise

The value of a solution is dened by comparing its strength with its cost effectiveness. Figure 6provides a graphical representation of the industry leaders evaluated, positioned in relation to bothcritical axes. Scores on the x axis reect Cost Efciency, which EMA considers to be a measure of

overall value. Products shown towards the right on this axis tend to be less expensive or have strongfeature/function in relation to overall cost. The y axis reects the vendors coverage of the specicfunctionality assessedin this case, APM for Cloud services. The size of the bubble represents a

relative measure of Vendor Strength.

General FindingsAll of the vendors in this category scored as Strong Value or better, which indicates that all exceeded

EMAs rigorous basic scoring criteria. As with the Suite solutions proled above, this does notindicate an overly rosy judgment on EMAs part, but is rather a manifestation of the phenomenon ofself-selection. Vendors not prepared to invest the resources required to traverse the Radar process,those daunted by the rigor of the survey and dialogue, and those less certain of product strength

declined to participate. From this perspective, those who did participate can thus be considered the

APM elite those providers who have invested in delivering complete solutions with substantialvalue to application management practitioners.

Netuitive, AppDynamics, and Aternity all scored in the Value Leader quadrant despite the fact thatthese are very different solutions. Netuitive is a top-level correlation solution, while AppDynamics is a

SaaS-based APM solution focused on code-level analytics. Aternity is an EUE solution focused on theendpoint, with rich capabilities supporting performance management, management of hosted desktopenvironments, and security compliance.

AppFirst, New Relic, Splunk, and INETCO all placed in the Strong Value category. Again, these arevery different solutions as well, but all support various elements of APM and related troubleshootingcapabilities. AppFirst and New Relic are SaaS-based, one focusing on Application Infrastructure

Management (AppFirst) and the other focusing on transaction tracing down to specic lines of code(New Relic). Splunk is a machine data aggregator and analyzer, while INETCO is well established inthe payments processing industry and in the process of extending capabilities to address the generalAPM market.

In-depth proles for all vendors in this report are listed in alphabetical order in the APM VendorProles section.


24/29


22



Value LeadersAppDynamics: AppDynamics Pro is specically designed for modern (complex, distributed)

application architectures in both the Cloud and the data center. Focused on deep transactionmonitoring at the application server level, AppDynamics supports both Java Virtual Machines (JVMs)

and Microsoft .NET Common Language Runtime (CLRs). AppDynamics also features a variety ofleading-edge automation capabilities and its IaaS automation is particularly noteworthy. It has provensupport for provisioning and de-provisioning large-scale, IaaS-based application server environments.

Aternity: With its Frontline Performance Intelligence Platform, Aternity has developed a distinctiveapproach to monitoring both Cloud and on-premise service performance, including that of hosteddesktop environments. By monitoring the endpoint and the next hop from the endpoint, Aternity

delivers a level of visibility to the experience of internal users that is difcult to match with otherEUE technologies. This technology can also be applied to a multiplicity of additional monitoringrequirements, such as identication of training issues, illicit usage, regulatory compliance, and levels of

services delivered by public Cloud vendors. Key differentiators include The Aternity FPI Platformssupport for hosted desktop environments and deep visibility to processes, keystrokes, and virtually allother events taking place at the user workstation.

Netuitive: Netuitive is a top-level, software-based analytics product that gathers, analyzes, andcorrelates metrics collected from enterprise source systems (sub-systems). It integrates with thirdparty management solutions by gathering/forwarding performance data to Netuitives Behavior

Learning Engine. The engine applies patented analytics to each data stream to learn normalbehavior, identifying anomalies, in many cases proactively, based on time-based trends such as dayof the week, month of the year, or season. Netuitives single pane of glass design delivers rst-ratetroubleshooting information and also extends the effective lifespan of management solutions already

in place.

Strong ValueAppFirst: AppFirst is a SaaS-based management solution focusing on application infrastructureperformance management. It is primarily an infrastructure monitoring solution with some top-levelcorrelation capabilities, supporting log le analytics, for example, and the ability to correlate metricsfrom a variety of sources. AppFirsts data collection technology allows continuous collection of metrics

from every process running on production servers. As a cost-effective product that is easy to deploy,this can be an excellent solution for smaller companies seeking basic application monitoring.

INETCO: INETCO has been a respected vendor in the payments industry since the 1980s, initiallyknown for its protocol conversion software. For the past ve years INETCO has built its businessaround Business Transaction Management (BTM) for complex tiered nancial applications. Eightypercent of INETCOs customers are service providers, primarily payment systems processors running

complex back end credit card processing systems. INETCO recently announced INETCO Insight5, a new solution intended to extend its business beyond the nancials vertical to cross-platformAPM. This is a distinctive offering that uses transaction tracing, network capture, and sophisticated

transaction templates to capture and identify transactions as they execute across tiered systems.INETCOs CloudTrails technology sees transaction performance across multiple off-premise tiers,with some visibility to whats happening IN the public Cloud as well. While INETCOs APM-specicfunctionality is still in the process of being built out, EMA welcomes INETCO to this new market.


25/29


23



New Relic: New Relic is part of a new breed of APM solutions founded by industry veterans with awealth of experience in the APM market. New Relic Pro is a SaaS-based solution delivered as a single,

all-inclusive product for development, application deployment, and IT operations teams. New RelicPro is optimized for application support professionals with some knowledge of application code, asit enables them to track application performance through servers and down to specic lines of code.

It delivers capabilities such as application performance analytics, Real User Monitoring, availabilitymonitoring, Server Monitoring, SQL performance, and web transaction tracing for applicationenvironments including Ruby, Java, .NET, PHP and Python.

Splunk: Splunk Enterprise is a multi-purpose machine data aggregation and assimilation solutionthat can be used to troubleshoot application problems, monitor application and system performance,provide visibility across the IT infrastructure, and investigate security incidents in both real-time and

historical time buckets. Splunk describes its offering as the engine for machine data and indicates thatcustomers can question, search, monitor and report on virtually any type of data (logs, events, metrics,status, alerts, traps, APIs, le system changes, diagnostic commands, etc.), and perform advancedanalytics on this data to provide operational intelligence. Splunk uniquely delivers a wide range of

features that lend themselves to APM monitoring and troubleshooting. It includes a exible, naturallanguage-like search function supporting analysis, trending, summarization, outlier detection etc. Inworking with Splunk customers, EMA has also seen signicant ROI from Splunks data management

functionality, which allows retention of large quantities of data with appropriate role-based accesscontrols and reduces storage requirements. Splunk can be deployed on premise, virtualized, or in theCloud (AWS, Rackspace, etc).

Exceptional Characteristics: AwardsEMA awards a limited number of citations to vendors with particularly noteworthy capabilities in one

or more functional areas. While the vendors participating in this particular study were all very strongand the choice was difcult, only six solutions received awards. Four were chosen from the Suite/Multi-Component group and two from the Point Solution group, commensurate with the numberof participants in each. These participants are worthy of special recognition for specic areas of

strength and/or unique areas of innovation.

AppDynamics: Best Infrastructure as a Service

(IaaS) AutomationAppDynamics Pro is a single component application performance

management solution specically designed for modern (complex,distributed) application architectures in both the Cloud and the datacenter. Focused primarily on application servers, AppDynamics has

visibility to both Java Virtual Machines (JVMs) and Microsoft .NET Common Language Runtime(CLRs). EMA is recognizing AppDynamics for its IaaS automation based on its support for provisioningand de-provisioning large-scale, IaaS-based application server environments.

One of AppDynamics key customers is a Cloud-based media vendor who is, in turn, one of thelargest customers of Amazons EC2 IaaS platform. The customer routinely creates 1,000 EC2 virtualmachines over 30-60 minutes to push new features as application code. Once the new environments

are in production, the vendor then deletes the 1,000 machines previously running that service.


26/29


24



While few, if any, enterprise IT organizations are leveraging dynamic EC2 environments to this extent,they may well do so in the future. From this perspective, AppDynamics support for public Cloud is

noteworthy and will likely be of interest to companies considering dynamic capacity management andCloud bursting.

Aternity: Most Comprehensive Endpoint

DiagnosticsBecause of its distinctive approach to End User ExperienceManagement, EMA is pleased to cite Aternity as the award winner forthe Most Comprehensive Endpoint Diagnostics. EMA research has

repeatedly found that companies leveraging Cloud services see EUEas a primary monitoring methodology for such environments. Monitoring the User experience is a failsafe way to determine the levels of overall service delivered to the desktop.

Aternity has developed a distinctive approach to monitoring both Cloud and on-premise serviceperformance, including that of hosted desktop environments. By monitoring the endpoint and thenext hop from the endpoint, Aternity delivers a level of visibility to the experience of internal

users that is difcult to match with other EUE technologies. This technology can also be applied toa multiplicity of related requirements, such as identication of training issues, illicit usage, regulatorycompliance, and levels of services delivered by public Cloud vendors.

eG Innovations: Best Private Cloud CoverageEMA sees eG Innovations heavy investments supporting virtualizedenvironments as worthy of an achievement award for Best Private

Cloud Coverage. This award is based on multiple functional capabilities,including:

Comprehensive Virtualization Coverage:VMware vSphere, Citrix XenServer, Microsoft Hyper-V,Microsoft Virtual Server 2005, Solaris LDOMs, Solaris Containers, AIX LPARs, RHEL, othersTBD.

Patented Virtualization/Cloud-aware Correlation:eG Innovations correlation supports the new sets ofdynamic inter-dependencies, such as VMware vMotions, that come with private Cloud.

Universal Agent Technologycontributes to cost and administration efciency. A single agent is capable

of monitoring over one hundred fty (150) applications, ten (10) different operating systems, andseven (7) different virtualization platforms.

In-N-Out virtual infrastructure monitoringsupports performance troubleshooting by providing both

depth and breadth of application visibility. It shows an outside view of a VM (the physical resources

the VM is consuming) in context to an inside view that indicates which application(s) inside theVM are causing the resource constraint. Both views are obtained using a single monitoring agent.


27/29


25



HP: Most Comprehensive and Real-Time

Service Model

A primary characteristic of leading-edge APM solutions is that theynot only report application performance, they support troubleshootingwhen performance degrades. For this reason, EMA sees real-time

topology models as key differentiators for companies leveraging Cloudservices. HP is particularly noteworthy in this regard, having invested heavily in developing a RunTime Service Model (RTSM) as an overlay to the HP Universal Conguration Management Database(UCMDB).

HPs RTSM correlates service models from multiple sources in the application stack to build a fullend- to- end view of the application operating environment. Leveraging metrics harvested from across

the execution ecosystem, HTTP\RMI tagging, and cross-VM correlations, HP discovers relationshipsamong application components and reports them to RTSM. This delivers a central, up-to-date model

of the application run time architecture, which in turn provides a data source for a Federated CMDB/CMS. EMA sees this as a visionary approach to delivering run-time topology, which is essential tomanaging, versus monitoring, applications in Cloud environments.

IBM: Best Cloud Vision and DesignEMA is pleased to award the Best Cloud Vision and Design awardto the IBM Tivoli Application Performance Management Solution.IBM scored above the norm on all ve axes measured. In addition togarnering the highest scores of any vendor, IBM has demonstrated

an in-depth understanding of the complex factors that supportCloud management. For example, IBM is one of few vendors paying attention to the CMDB/CMSimplications of Cloud. As application components increasingly span on-premise, public Cloud, and

partner Clouds, conguration tracking will be a critical capability for performance management andapplication troubleshooting.

IBM is also actively addressing Cloud capacity planning within IBM Smart Cloud Monitoring.Capabilities include performance trending to proactively predict resource bottlenecks, what-if analysisto determine the number of workloads that can be added to existing resources, and optimization ofworkload placement. IBM has good coverage of virtualization platforms, notably including Red Hat,

z, and Solaris, which are missing in most competing solutions.

IBM is one of few vendors who have thought through monitoring and management of complex

Cloud transactions such as SaaS to SaaS integrations. IBM is also noteworthy in terms of its overallfocus on automation. For example, although IBM has synthetic transaction technology, this is not the

centerpiece of its Cloud APM strategy, as it is for some competing vendors. Instead, IBM has focusedon analytics, including tracing and stitching capabilities that reduce requirements for ongoing hands-on

administration. In terms of cost efciency, IBM has also added capabilities to make its production-grade products more easily consumable by the mid-market.


28/29


26



OPNET: Best Overall AutomationOPNET APM Xpert Suite is a comprehensive solution for

managing the performance of applications in physical, virtual, andCloud environments. OPNETs High Denition Performance

Management approach combines broad visibility across domains withgranular drill-down within each domain. It heavily leverages analyticsto automate troubleshooting and root cause diagnosis.

As the hallmark of modern applications is their integrated nature, EMA analysts see the network,supported by leading edge analytics, as the Rosetta Stone reconciling top-down, ow-basedapplication/transaction insight with the bottom-up view of infrastructure topology. Without near

real-time correlation of both in context to one another, Application Performance Management (versusApplication Performance Monitoring) cannot be automated.

OPNET has evolved its solution set based on a similar vision and has continued to add features

accordingly. OPNETs Federated Analytics provide a good example of a new breed of solutionscapable of knitting together multiple information sources, including transaction ow, into real time

models of application execution. Such automation supports troubleshooting and fast Mean Time toResolve (MTTR) in the short term. In the longer term, it builds a foundation for next generation APMcapabilities that will someday enable lights out enterprise management to become a reality. OPNETviews Cloud as a continuum, with hybrid as the nal answer and is evolving the toolset accordingly.


29/29

About Enterprise Management Associates, Inc.

Founded in 1996, Enterprise Management Associates (EMA) is a leading industry analyst firm that provides deep insight across the fullspectrum of IT and data management technologies. EMA analysts leverage a unique combination of practical experience, insight intoindustry best practices, and in-depth knowledge of current and planned vendor solutions to help its clients achieve their goals. Learnmore about EMA research, analysis, and consulting services for enterprise line of business users, IT professionals and IT vendors at

www.enterprisemanagement.com or blogs.enterprisemanagement.com. You can also follow EMA onTwitter or Facebook.

This report in whole or in part may not be duplicated, reproduced, stored in a retrieval system or retransmitted without prior written permission ofEnterprise Management Associates, Inc. All opinions and estimates herein constitute our judgement as of this date and are subject to change without notice.Product names mentioned herein may be trademarks and/or registered trademarks of their respective companies. EMA and Enterprise Management

Associates are trademarks of Enterprise Management Associates, Inc. in the United States and other countries.

2012 Enterprise Management Associates, Inc. All Rights Reserved. EMA, ENTERPRISE MANAGEMENT ASSOCIATES, and the mobius symbolare registered trademarks or common-law trademarks of Enterprise Management Associates, Inc.

Corporate Headquarters:

5777 Central Avenue, Suite 105Boulder, CO 80301
http://www.enterprisemanagement.com/http://blogs.enterprisemanagement.com/http://twitter.com/ema_researchhttp://www.facebook.com/pages/Enterprise-Management-Associates-EMA/232626436757941http://www.facebook.com/pages/Enterprise-Management-Associates-EMA/232626436757941http://twitter.com/ema_researchhttp://blogs.enterprisemanagement.com/http://www.enterprisemanagement.com/

ema apm 2012 radarreport summary

Documents