User Survey Analysis: Customers Rate Their CPM Vendors, 2012

4 June 2013 ID:G00247384

Analyst(s): Christopher Iervolino, John E. Van Decker

VIEW SUMMARY

Gartner's survey of 275 customer references from 13 corporate performance management vendors

between September and November 2012 has produced valuable insights and benchmarking data

for IT leaders assessing CPM solutions. Don't just focus on the megavendors when evaluating

prospective providers.

Overview

Key Findings

Among the megavendors, SAP has improved overall satisfaction ratings in relation to

Oracle, and IBM has maintained an above-average overall rating; however, survey results

highlight room for improvement for all three providers.

Annualized cost comparisons indicate that software-as-a-service solutions are not

significantly less expensive than many traditional on-premises offerings. For both options,

business value attained and total cost of ownership should primarily guide buying

decisions.

Results indicate distinctive customer satisfaction differences among corporate performance

management vendors in vendor-, product- and implementation-related scores. Statistics

related to average number of users per implementation, popularity of modules, length of

time used and cost also characterize vendor solutions.

Many specialist CPM vendors score consistently higher than the megavendors, which

shows they have credible solutions, despite their relatively small size and market share.

Recommendations

Extend product evaluations past the assessment of individual functions, and consider the

cost and complexity of implementations and ongoing use, as well as the overall value of

the vendor relationship.

Conduct cost comparisons of SaaS versus on-premises solutions that consider a wide

range of TCO factors, including ongoing consulting, IT support and upgrade expenditures.

Consider specialist CPM vendors in assessments, and do not focus only on the

megavendors.

TABLE OF CONTENTS

CONTENTS

Survey Objective

Data Insights

o Vendor Experience Ratings

o Product Experience Ratings

o Implementation Experience Ratings

o Solution Usage Comparisons

o Solution Cost Comparisons

o Methodology

TABLES

Table 1.

Response by Vendor Customer Locations

Table 2.

Estimated Number of Years Solution Has Been in Use

FIGURES

Figure 1.

Overall Satisfaction

Figure 2.

Overall Satisfaction Versus Average Implementation Size

Figure 3.

Vendor Experience

Figure 4.

Overall Satisfaction Versus the Top-Three Differentiating Vendor Attributes

Figure 5.

Product Experience

Figure 6.

Overall Satisfaction Versus the Top-Three Differentiating Product Attributes

Figure 7.

Implementation Ratings

Figure 8.

Overall Satisfaction Versus the Top-Three Differentiating Implementation Attributes

Figure 9.

Strategic CPM Usage

Figure 10.

Module Usage

Figure 11.

Cost Characteristics: Total Cost to Date

Figure 12.

Annual Cost Characteristics

Figure 13.

Organization Size (Number of Employees)

Figure 14.

Geographic Composition

Figure 15.

Industry Composition

Survey Objective

This document was revised on 6 June 2013. For more information, see the Corrections page.

Toward the end of 2012, Gartner conducted its annual corporate performance management (CPM)

Magic Quadrant survey.1 In that survey, 275 CPM customers from 13 different vendors shared

valuable insights for companies selecting CPM vendors and for customers benchmarking suppliers.

The detailed results provide additional insights to support prospective customers in fine-tuning

their RFPs, product selection criteria and project plan approaches for implementations and

upgrades. This survey comprised a key component of the CPM Suite Magic Quadrant report (see

"Magic Quadrant for Corporate Performance Management Suites"). The analysis of these detailed

results is an important complement to that study.

Table of Contents

Data Insights

This research reveals customer satisfaction ratings as they relate to vendor, product and

implementation attributes. It uses the three scores from each category with the greatest variances

among vendors to highlight the most-differentiating factors. It then evaluates responses alongside

the average number of users per implementation, and the levels of use for each module in the

vendors' product suites. This research also sheds light on other differentiating solution cost

characteristics.

Table 1 shows the number of responses by the vendor and describes their customer locations.

Table 1. Response by Vendor Customer Locations

Vendor No. of Survey Participants Customer Locations

IBM 30 Worldwide

Prophix 29 Mainly North America, but also worldwide

Board International 26 Mainly EMEA, but also worldwide

Host Analytics 24 Mainly North America, but also worldwide

Longview 24 Mainly North America, but also worldwide

Oracle 23 Worldwide

Tagetik 21 Mainly EMEA, but also worldwide

Bitam 19 Mainly North and South America, but also worldwide

KCI Computing 17 Mainly North America, but also worldwide

SAP 17 Worldwide

SAS 17 Worldwide

Infor 15 Worldwide

prevero 13 Mainly EMEA, but also worldwide

Source: Gartner (June 2013)

Table of Contents

Overall, most survey participants were satisfied with their CPM vendors (see Figure 1). On a scale

of 1 to 7, the average rating was 6.09. This favorable rating, which is higher than those in previous

years, indicates that CPM solutions are mature and are delivering value. However, the standard

deviation was fairly high (0.94), which reflects a greater variation in scores, especially for KCI

Computing (standard deviation = 1.46) and prevero (standard deviation = 1.38). Most of the

customers chosen by the vendors were satisfied with their solutions; however, the level of

variability underscores the importance of matching the correct CPM vendor, product and

implementation capabilities to a given set of organizational needs.

Figure 1. Overall Satisfaction

The rating is an average of respondents' overall aggregate score by vendor. The figure represents

customer perceptions, not Gartner's opinion. The graph may feature vendors that in Gartner's

opinion don't deliver the benefits described (N = 275).

Source: Gartner (June 2013)

Table of Contents

Although it's reasonable to assume that vendors with smaller implementations and few users

would consistently score higher in questions related to vendor support, these survey results do not

wholly support this. As in 2011, vendors Tagetik and Longview score above-average satisfaction

ratings and have an above-average number of users per implementation. IBM also scored above

average in both categories; however, Tagetik and Longview's higher scores are standouts in this

area (see Figure 2).

Figure 2. Overall Satisfaction Versus Average Implementation Size

Source: Gartner (June 2013)

Table of Contents

The overall customer satisfaction ratings are an aggregation of vendor, product and

implementation scores. To gain insights into these overall ratings, it's necessary to drill down into

more-detailed survey results that illustrate more-specific strengths and weaknesses of each

vendor.

Table of Contents

Vendor Experience Ratings

The following information highlights results related to the vendor experience, such as sales,

support and ongoing communication. The survey asked this question regarding the vendor

experience: "Compared with experiences with other vendors, how was the experience with this

vendor on the following attributes?"

Ongoing support and after-sales care

Overall value for the money

Ongoing communication

Responsiveness of telephone support

Experience throughout the sales process

Additional implementation and deployment

Ease of applying upgrades, fixes and patches

Professional service implementation, and deployment training and handover

The attributes that resulted in the largest variations between high and low scores have the biggest

impact on the overall client satisfaction score, and contribute most to differentiating the vendors

within this category. The three attributes with the greatest variances, and their respective scores

for each vendor, are represented in Figure 3. The question asked was, "Compared with

experiences with other vendors, how was the experience with this vendor on the following

attributes?"

Figure 3. Vendor Experience

The figure represents customer perceptions, not Gartner's opinion, and may feature vendors that

in Gartner's opinion don't deliver the functional capability described (N = 275).

Source: Gartner (June 2013)

Table of Contents

Figure 4 shows overall customer satisfaction, compared with the top vendor-differentiating

attributes.

Figure 4. Overall Satisfaction Versus the Top-Three Differentiating Vendor Attributes

The figure represents customer perceptions, not Gartner's opinion, and may feature vendors that

in Gartner's opinion don't deliver the functional capability described (N = 275).

Source: Gartner (June 2013)

Table of Contents

Similar to last year, Longview and Tagetik scored well in this category, and Board International's

rating is also on a par with these vendors. Of the three, Tagetik's high rating stands out, given its

above-average revenue growth rate of 25.7% in 2012. Host Analytics, which had ranked among

these vendors in previous years, scored the highest in 2012. As a pure software as a service

(SaaS) vendor, it naturally scored the highest rating for "ease of applying

upgrades/patches/fixes"; however, it also scored highest on the other two differentiating ratings,

including the important "overall value for the money" rating. Host Analytics has achieved this score

during a period of high revenue growth (78.3%), which indicates its growing organizational

maturity. (All vendor revenue growth figures can be found in "Market Share Analysis: Business

Intelligence, Analytics and Performance Management, Worldwide, 2012.")

The largest variation of all vendor, product and implementation ratings existed for "ease of

applying upgrades/patches/fixes." This suggests that the greatest single difference between the

vendors, from the customer's perspective, relates to upgrades, which encompasses ongoing

maintenance and support cost sensitivity, as well as the desire for new product features available

in more-current software versions.

Consistent with previous years, all three megavendors (IBM, Oracle and SAP) scored below

average in this vendor-specific category. Although Oracle scored the lowest in average, it also had

the lowest "overall value for the money" rating. This low rating may have been affected by the

increased awareness of less-expensive CPM options in the marketplace. Slowed CPM market

growth figures for 2012 seem to indicate that additional cost pressures are affecting CPM vendors

differently; for example subscription-based options, such as Host Analytics, have increased their

revenue in 2012 at higher-than-average rates. On the whole, Oracle's average score was most

affected by "responsiveness of telephone support," where it received the lowest score of any other

vendor rating.

All three of the highest variations in overall vendor rating are related to cost. Furthermore, the

largest variations occurred for "ease of applying upgrades/fixes/patches" and "responsiveness of

telephone support." These ratings relate to ongoing solution effort and cost, indicating a growing

gap in customer sentiment related to ease of use and cost considerations among the different

vendors. Prospective customers should note the correlation of these vendor-specific criteria to

overall customer satisfaction. The horizontal distance from average is an indication of how these

vendor-specific factors have affected their overall customer satisfaction ratings.

Table of Contents

Product Experience Ratings

The survey asked the following question regarding product experience: "To what extent has the

organization's vendor CPM solution met the following requirements?"

Improved the quality, accuracy and timeliness of financial and management reporting

Added more sophistication and confidence to the budgeting, planning and forecasting

(BP&F) processes

Reduced the effort to produce budgets, and reduced the budget cycle time

Allowed business users to get the information they need, with reduced reliance on IT

Provided a better understanding of current performance

Allowed a better prediction of financial and business performance

Provided a linkage between strategic planning, financial budgets and operational activity

Allowed a better understanding of profitability drivers

Improved the ability to meet compliance requirements, including external disclosure

As with responses related to vendor experience, the attributes that resulted in the largest

variations between high and low scores have the biggest impact on the overall client satisfaction

score, and contribute most to differentiating the vendors within this product experience category.

The three attributes with the greatest variances in this category and their respective scores for

each vendor are represented in Figure 5. The question asked was, "To what extent has the

organization's vendor CPM solution met the following requirements?"

Figure 5. Product Experience

Source: Gartner (June 2013)

Table of Contents

Compared with last year's results, Host Analytics and Board International scored notably higher.

This indicates a growing satisfaction with available product functionality in these areas and an

increased sophistication of use for these vendors' solutions (see "User Survey Analysis: Customers

Rate Their Corporate Performance Management Vendors"). These ratings indicate that the widest

range of end-user satisfaction exists for some key product functionality —compliance, giving

insight into the drivers of profitability and providing a linkage between strategic financial planning,

budgeting and operational planning. The rating for Host Analytics is somewhat surprising in that it

is newer to the market than most of the others. A reasonable assumption is that Host Analytics

does not have the breath of functionality of some of the more widely used solutions that have been

around longer; however, these ratings are a measure of satisfaction with ease of use (accessibility

of the functionality), as well as satisfaction with the functionality itself (robustness of the

functionality). Vendors that scored well in this category are likely to be able to attribute their

success to customer satisfaction with a combination of feature accessibility and capability (see

Figure 6).

Figure 6. Overall Satisfaction Versus the Top-Three Differentiating Product Attributes

Source: Gartner (June 2013)

Table of Contents

The horizontal distance from average is an indication of how these product-specific factors have

affected their overall customer satisfaction ratings. Of the three satisfaction categories surveyed

(vendor, product and implementation), the most narrow range of scores exists for product ratings

(between 4.63 and 5.72). That is, vendor- and implementation-related areas had the greatest

impact on overall satisfaction scores. This indicates that customers are more satisfied with the

product-specific functionality their solutions provide than with vendor- and implementation-specific

items. As a result, prospective customers should ensure that they evaluate new solutions from a

product functionality perspective, as well as in terms of information gathered from vendor

references regarding their vendor and implementation experiences.

Table of Contents

Implementation Experience Ratings

The survey requested the following information regarding each customer's implementation

experience by requesting its level of agreement with the following statements about the

organization's implementation of the vendor's CPM solution:

The vendor's professional service staff had the skills and knowledge to make the project a

success.

The CPM applications delivered the functionality expected.

The anticipated business benefits from implementing the CPM applications were realized.

It was easy to adapt the CPM applications to meet business requirements over time.

Users found the system easy to use and manage.

IT finds the CPM application technology straightforward to maintain and support.

The solution gained wider acceptance than expected.

The implementation was easier than expected.

The implementation was less expensive than expected.

Because the attributes that resulted in the largest variations between high and low scores have the

biggest impact on the overall client satisfaction score, and contribute most to differentiating the

vendors within this product-experience-related category, the three attributes with the greatest

variances within this category and their respective scores for each vendor are examined in Figure

7. The request was, "Indicate agreement with the following statements about the organization's

implementation of the vendor's CPM solution."

Figure 7. Implementation Ratings

Source: Gartner (June 2013)

Table of Contents

Some of these ratings are expected — for example, it's natural that Host Analytics would score

highest for ease of IT maintenance and support, because it's a SaaS solution. Interestingly, Board,

which does not offer a SaaS solution, and Tagetik, whose customer respondents did not use its

cloud-based offering, were not far behind. This shows that CPM apps are different from ERP and

other business applications that require heavy IT support; IT complexity should not be a forgone

conclusion.

Another significant expectation-related result here is that every vendor had its lowest rating for the

item, "our implementation was cheaper than expected." Although installation costs varied among

the vendors, this indicates a consistent set of false expectations with CPM solutions. This may

occur during the sales or the implementation scoping or a combination of the two. This result also

reflects the importance of cost as a satisfaction factor. Prospective customers should ensure they

include a rigorous customer reference effort that includes an exploration of consulting and support

fees as a necessary component of any CPM purchase or project scoping exercise (see Figure 8).

Figure 8. Overall Satisfaction Versus the Top-Three Differentiating Implementation Attributes

Source: Gartner (June 2013)

Table of Contents

The tight diagonal arrangement indicates close relationship between implementation and

satisfaction. This is correlation is highest for these implementation ratings, underscoring the

importance of considering implementation project needs alongside the software purchase when

selecting a particular vendor's solution.

Table of Contents

Solution Usage Comparisons

Although most CPM implementations take advantage of office-of-finance CPM capability initially,

organizations can realize significant improvements by taking advantage of strategic CPM. (see

"Getting More Value From CPM: Strategic Versus Office-of-Finance CPM"). This graphic is one

indication of the degree to which different vendor's solutions are used strategically. Figure 9

illustrates the relationship between overall satisfaction and two key metrics that indicate a more-

strategic use of each vendor's CPM solution. The first is the average number of users per

implementation, and the second is module use, which is represented by circle size (more

information regarding module use can be found in Figure 10).

Figure 9. Strategic CPM Usage

Source: Gartner (June 2013)

Table of Contents

Figure 10. Module Usage

Source: Gartner (June 2013)

Table of Contents

Those vendors with a larger number of users per implementation (distance to the right) and a

larger circle in Figure 9 (more widely used set of CPM modules) indicate more strategic CPM use.

Those that have also scored high levels of satisfaction (vertical position) are able to do so with

more-complex installations that support a wider number of CPM processes.

How many CPM modules are used is an indication of the popularity of the each of the vendors'

modules. The survey asked the question, "How many users does the organization currently have

for each of the following features of the vendor's CPM solution?" The answers were converted into

percentages of the number of overall users for each vendor to provide consistent comparison

mechanism among vendors. Total scores were ranked from highest to lowest.

This information was then compared with the average number of users per implementation,

another measure of CPM maturity, as well as the average customer satisfaction rating. By

comparing how extensively the product is used (by estimating the modules used and number of

users) the average level of CPM maturity can be estimated. Furthermore, by comparing these

statistics against overall satisfaction, the ease in which the solution allows for additional CPM

maturity for implementations of different sizes can be roughly estimated.

Although the customer sample size is small, compared with the total number of customers (this is

especially true for IBM, Oracle and SAP), the survey data indicates patterns of usage among

survey respondents, such as:

IBM's solutions were reported to be used for a broader set of CPM processes than were

SAP's (and somewhat more than Oracle's as well). IBM also had a higher overall

satisfaction rating than either Oracle or SAP.

Longview and Tagetik were able to maintain high customer satisfaction ratings, even with

a high number of average users. Also, a lower percentage of Tagetik customer

respondents use their applications for BP&F, as compared with Longview's.

Prophix, Board International, SAS Institute and Host Analytics respondents support a wide

array of CPM processes, although each does so for implementations of different average

size and, most likely, of varied complexity.

Table of Contents

Solution Cost Comparisons

Respondents were asked to provide the approximate total cost to date of their organizations' CPM

solutions for each of five cost categories. This included subscription fees; external services

(including consulting, system integration, managed services, hosting, and outsourced process);

hardware; software; and internal labor and head count. Because IBM and KCI Computing each had

one customer response with very high costs, median results were used, instead of averages, to

reduce the impact of these two outliers. The breakdown of costs by vendor is shown in Figure 11.

Figure 11. Cost Characteristics: Total Cost to Date

Source: Gartner (June 2013)

Table of Contents

These results clearly indicated that Oracle's CPM implementations are the most expensive,

especially for external services, among survey respondents. However, Oracle's CPM solutions have,

on average, been used for longer periods of time. Table 2 shows the median number of years that

respondents reported their solution has been in use (if they have multiple solutions from the

vendor, the solution that was used the longest was reported on).

Table 2. Estimated Number of Years Solution Has Been in Use

Vendor Years Used

Oracle 8

Longview 7

SAS Institute 6

Table 2. Estimated Number of Years Solution Has Been in Use

Vendor Years Used

IBM 5

KCI Computing 5

Bitam 4

Board International 4

Infor 3

prevero 3

Tagetik 3

SAP 2.5

Host Analytics 2

Prophix 2

Source: Gartner (June 2013)

Table of Contents

Clearly, vendors with customers that have used their solutions for longer periods of time (such as

those using Longview and SAS Institute's solutions) now show lower comparable costs, because

they have had more years over which to spread initial software purchase and implementation

costs. Others that, on average, have relatively new installations, such as those using SAP's and

Host Analytics' solutions, have higher per-year costs. Oracle remains the most expensive, both on

a total and per year basis; however, it's important to consider that Oracle's customer respondents

also estimated a relatively high number of average users. On a per-user basis Oracle's average

costs per year are lower, as are Longview's and Tagetik's. These annualized cost comparisons also

illustrate ongoing costs over time — for example, Figure 12 highlights the higher annual cost of

external services for both SAP and Oracle.

Figure 12. Annual Cost Characteristics

Source: Gartner (June 2013)

Table of Contents

Host Analytics, a SaaS vendor, has a relatively low initial cost; however, when these costs are

annualized, they are higher than many on-premises vendors. Also, because Host Analytics'

respondents reported a relatively low number of users, costs per year, per user are also higher

relative to many of the other solutions. Regarding SaaS versus on-premises cost comparisons, this

survey was generally completed by finance users and internal labor and head count estimates may

not always reflect accurate ongoing IT support costs. In addition, ongoing costs, such as software

upgrades, are generally built into multitenant SaaS vendor costs; such is the case with Host

Analytics. However, these results suggest that prospective customers should look past short-term

costs and perform on-premises versus SaaS cost comparisons from a total cost of ownership (TCO)

perspective over the anticipated life of the solution. Customers should consider all cost

considerations including ongoing internal and external consulting expenses and both long-term

subscription and license fees when selecting a CPM vendor.

On the whole, this more-detailed analysis of survey results illustrates the credibility of smaller CPM

vendors across a number of areas for larger, more-complex CPM solutions or more-focused ones.

A number of these vendors consistently score high marks in key satisfaction areas, such as overall

value for the money and ease in adapting the solution for changing business requirements. Despite

the consistent leader ratings of the megavendors in the Magic Quadrant, smaller vendors are

highly credible options for many CPM needs and should be considered in a wide array of product

evaluations. In addition, survey results regarding average implementation size, modules used,

length of use and cost can help determine correct product fit and guide product selection

processes.

Table of Contents

Methodology

As part of our research for the CPM Magic Quadrant, Gartner conducted an English-language Web

survey of 275 CPM suite customers (up from 233 in 2011). The individuals surveyed were

nominated by CPM vendors at Gartner's request (Magic Quadrant methodology mandates

gathering data from vendor customers). The survey was composed of 21 questions regarding

customer experiences in working with vendors. Gartner requested contacts knowledgeable about

how the vendors' CPM products were used by customer organizations. The vendors did not see the

questionnaire before the customers were nominated, and responders were made aware that their

answers would remain strictly anonymous. The survey took 20 minutes and covered the

customer's use of the nominating vendor's CPM solution.

Gartner expected that, because the survey participants were vendor reference customers, the

results would reflect a more positive experience of using and implementing each vendor's CPM

solutions. However, the relative comparison of results is valid, since all respondents were

nominated by their respective vendors. Although representative of this group, and providing useful

indications of customer experiences and deployment characteristics, the survey results may not

reflect the views of the general population using these products.

Cost characteristics were calculated by converting rates to U.S. dollars as of the last day of the

survey, 13 November 2012. Median results were then selected to reduce the effect of outliers.

Size, geography and industry characteristics are shown in Figure 13, Figure 14 and Figure 15,

respectively.

Figure 13. Organization Size (Number of Employees)

Source: Gartner (June 2013)

Table of Contents

Figure 14. Geographic Composition

Source: Gartner (June 2013)

Table of Contents

Figure 15. Industry Composition

Source: Gartner (June 2013)

Table of Contents

EVIDENCE

1 This annual survey was conducted from early September until the middle of November 2012. A total of 275

completed surveys were returned. Additional details regarding this survey can be found in "Magic Quadrant for

Corporate Performance Management Suites."

© 2013 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of

Gartner, Inc. or its affiliates. This publication may not be reproduced or distributed in any form

without Gartner’s prior written permission. If you are authorized to access this publication, your use of

it is subject to the Usage Guidelines for Gartner Services posted on gartner.com. The information

contained in this publication has been obtained from sources believed to be reliable. Gartner disclaims

all warranties as to the accuracy, completeness or adequacy of such information and shall have no

liability for errors, omissions or inadequacies in such information. This publication consists of the

opinions of Gartner’s research organization and should not be construed as statements of fact. The

opinions expressed herein are subject to change without notice. Although Gartner research may

include a discussion of related legal issues, Gartner does not provide legal advice or services and its

research should not be construed or used as such. Gartner is a public company, and its shareholders

may include firms and funds that have financial interests in entities covered in Gartner research.

Gartner’s Board of Directors may include senior managers of these firms or funds. Gartner research is

produced independently by its research organization without input or influence from these firms, funds

or their managers. For further information on the independence and integrity of Gartner research, see

“Guiding Principles on Independence and Objectivity.”

About Gartner | Careers | Newsroom | Policies | Site Index | IT Glossary | Contact Gartner